Splunk

Expand messages by default

https://answers.splunk.com/answers/103005/expand-json-messages-by-default.html

At the top of your search results are field names, above the Time field name is a paintbrush with the word Format next to it. Click on this and select All lines for the Max Lines setting and Full for the Click Selection setting. Enjoy.

http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Rex

source="cisco_esa.txt" | rex field=_raw "From: <(?<from>.*)> To: <(?<to>.*)>" | dedup from to | table from to

http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/Stats

Simple: stats (stats-function(field) [AS field])... [BY field-list]

Complete: stats [partitions=<num>] [allnum=<bool>] [delim=<string>] ( <stats-agg-term>... | <sparkline-agg-term>... ) [<by-clause>]

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Highlight
| highlight login,logout

https://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/Fields
https://docs.splunk.com/Documentation/SplunkCloud/7.0.0/SearchReference/Dedup

Remove duplicates of results with the same 'host' value.

... | dedup host

Example 2:

Remove duplicates of results with the same 'source' value and sort the events by the '_time' field in ascending order.

... | dedup source sortby +_time

Example 3:

Remove duplicates of results with the same 'source' value and sort the events by the '_size' field in descending order.

... | dedup source sortby -_size

Example 4:

For events that have the same 'source' value, keep the first 3 that occur and remove all subsequent events.

... | dedup 3 source

Example 5:

For events that have the same 'source' AND 'host' values, keep the first 3 that occur and remove all subsequent events.

... | dedup 3 source host

Specify other fields that are relevant. The most common fields to specify are sourcetype and host.
Sharing results with others
Save as, export

Saving searches for reuse
Creating alerts from searches

Commands can do anything to the events that they are handed. Usually, a command does one of the following:
Modifies or creates fields—for example, eval, rex
Filters events—for example, head, where
Replaces events with a report—for example, top, stats
Sorts the results of a search—using sort
Some commands can act as generators, which produce what you might call synthetic events, such as |metadata and |inputcsv.

| top date_month date_wday
Using timechart to show values over time
| timechart count by date_wday
There are a number of commands that create new fields, but the most commonly used are eval and rex.

index=_internal sourcetype=splunk* | top limit=5 name | sort - name

Time modifiers
earliest=-15m latest=now
| timechart count span=5m

https://docs.splunk.com/Splexicon:Timerangepicker

AND OR NOT ()
| stats count by method
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/CommonStatsFunctions
avg(X): Returns the average of the values of the field X
dc(X): Returns the count of distinct values of the field X
max(X): Returns the maximum value of the field X
min(X): Returns the minimum value of the field X
perc<X>(Y): Returns the Xth percentile of the field X, for example perc95(X)
sum(X): Returns the sum of the values of the field X

| top url limit=5 showperc=false
but use rare instead of top. The term rare will find those events that are the most unlikely ones.

For all basic purposes, you can use stats and chart interchangeably. However, there will be differences in how stats and chart group data together.
| chart count by method url
| stats count by method url

The timechart command, on the other hand, creates a time series chart with statistical aggregation of the indicated fields. This command is widely used when creating different types of chart.

 | timechart span=15m count by url
Visualizations

Search command - eval

The rex or regular expression command is extremely useful when you need to extract a field during search time that has not already been extracted automatically. The rex command even works in multi-line events

index=main | rex field=http_user_agent
     "Chrome/(?<Chrome_Version>.+?)?Safari" | top Chrome_Version

If you want your searches to be faster, use Fast Mode. Fast mode will not attempt to generate fields during search time, unlike the default smart mode. This is very good to use when you do not know what you are looking for. Smart Mode looks for transforming commands in your searches. If it finds these, it acts like fast mode; if it doesn't, then it acts like verbose mode.

Event Sampling
Subsearch

Including clear key-value pairs
if it is important to include spaces in the values, in text fields, for example, you should surround the value with quotes:
key1="value1" or user="Matt Nguyen"

Time
http://docs.splunk.com/Documentation/Splunk/7.0.1/SearchReference/SearchTimeModifiers
earliest=-60m
To search for errors affecting bob in the last 3 hours, snap to the beginning of the hour using earliest=-3h@h bob error
To search for errors affecting bob yesterday, use earliest=-1d@d latest=-0d@d bob error
To search for errors affecting bob since Monday midnight, use earliest=-0@w1 bob error

You cannot use different time ranges in the same query

http://docs.splunk.com/Documentation/Splunk/7.0.1/SearchReference/SearchTimeModifiers

The syntax for using time modifiers is [+|-]<time_integer><time_unit>@<time_unit>

For example, to start your search an hour ago, use either of the following time modifiers.

earliest=-h

or

earliest=-60m

3. Search with an exact date as a boundary

With a boundary such as from November 5 at 8 PM to November 12 at 8 PM, use the timeformat %m/%d/%Y:%H:%M:%S.

earliest="11/5/2017:20:00:00" latest="11/12/2017:20:00:00"

4. Specify multiple time windows

You can specify multiple time windows using the timeformat %m/%d/%Y:%H:%M:%S. For example to find events from 5-6 PM or 7-8 PM on specific dates, use the following syntax.

(earliest=“1/22/2018:17:00:00" latest="1/22/2018:18:00:00") OR (earliest="1/22/2018:19:00:00" latest="1/22/2018:20:00:00")

https://answers.splunk.com/answers/59305/how-to-find-records-that-do-not-contain-a-certain-field.html
sourcetype="my sourcetype" someSearchTerms NOT User=*

sourcetype="my sourcetype" someSearchTerms | where isnull(User)
https://answers.splunk.com/answers/476391/how-to-list-all-values-of-an-extracted-field.html
base search | stats values(yourfield)
base search | table fieldName | dedup fieldName
base search | stats count by fieldName
https://answers.splunk.com/answers/424088/how-to-see-all-values-in-a-field.html
How to see all values in a field

 ... | top a_number SourceName

http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchReference/ListOfSearchCommands
https://answers.splunk.com/answers/82066/how-to-make-the-search-in-ascending-order.html

Well, the way Splunk works on, produces and displays search results is in reverse time order. That is, the latest events/results show up first - or on top of the list.
To reverse this order, use the reverse command. But note, that this only changes the way the events/results are displayed - it does not change the way the results/events are searched and generated.

Ex.

index=my_index sourcetype=my_sourcetype earliest=-5m | reverse

http://geek-university.com/splunk/sort-command/
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort

The sort command sorts all of the results by the specified fields. Results missing a given field are treated as having the smallest or largest possible value of that field if the order is descending or ascending, respectively.

sort [<count>] <sort-by-clause>... [desc]

Required arguments

<sort-by-clause>

Syntax: ( - | + ) <sort-field>, ( - | + ) <sort-field> ...

Description: List of fields to sort by and the sort order. Use a minus sign (-) for descending order and a plus sign (+) for ascending order. When specifying more than one field, separate the field names with commas. See Sort field options.

Sort results by "ip" value in ascending order and then by "url" value in descending order.

... | sort num(ip), -str(url)

Example 2:

Sort first 100 results in descending order of the "size" field and then by the "source" value in ascending order. This example specifies the type of data in each of the fields. The "size" field contains numbers and the "source" field contains strings.

... | sort 100 -num(size), +str(source)

Example 3:

Sort results by the "_time" field in ascending order and then by the "host" value in descending order.

... | sort _time, -host

Example 4:

Change the format of the event's time and sort the results in descending order by the Time field that is created with the eval command.

... | bin _time span=60m | eval Time=strftime(_time, "%m/%d %H:%M %Z") | stats avg(time_taken) AS AverageResponseTime BY Time | sort - Time

Events before or after X seconds
- Need manually edit the url to remove unneeded queries

https://answers.splunk.com/answers/136664/changing-max-length-of-field.html

You might be hitting this limit (from limits.conf):

 maxchars = <integer>
 * Truncate _raw to this size and then do auto KV.
 * Defaults to 10240 characters.

I have a field that is more than 10,000 characters. I updated props.conf to include

 [source::log.txt]
 TRUNCATE=20000

http://dev.splunk.com/view/logging/SP-CAAAFCK

One of the most powerful features of Splunk software is its ability to extract fields from events when you search, creating structure out of unstructured data. To make sure field extraction works as intended, use the following string syntax (using spaces and commas is fine):

key1=value1, key2=value2, key3=value3 . . .

Splunk software's automatic field extraction is worth the size difference.

Splunk software knows how to index. It will catch up where it left off so you won't lose logging data.

Interesting Fields

https://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Search/ExportdatausingSplunkWeb

When you export large amounts of data using the Export button, the session might timeout before the export is complete. You can extend the session timeout limit.

1. Click Settings > Server Settings > General Settings.
2. In the Splunk Web section, increase the number in the Session timeout field.
3. Click Save.