Filter lines in log file with ERROR

A couple of days ago, a friend of mine was interested to know how to filter a log file (in my case a log4j log file) for ERRORs alone.

An hour of tricks sharing followed and here is the gist of the conversation that you will be interested in.

Find me wherever I am

Listing all the lines in the log file which has occurrences of ERROR is as simple as executing the following command

grep "ERROR" <YOUR LOG FILE NAME>

When there is a beginning

And if you are interested in the lines which start with ERROR, then it just involves the usage of the regular expression for "begin"

grep "^ERROR" <YOUR LOG FILE NAME>

...there's an end

A similar trick applies if you are interested in searching lines which has occurrences of "Exception" at the end of the line (specially in case of Java stack traces), then you would use

grep "Exception$" <YOUR LOG FILE NAME>

Don't be so sensitive

Good news is that you need not be case sensitive to grep with regard to search string. You could just use

grep -i "exception$" <YOUR LOG FILE NAME>

(i as in insensitive)

Second thoughts

That said, imagine you are printing the username or a unique trace id of your request in your logs and you would want to filter your log based on that. And suppose this username/trace id is located as the 3rd word in every line. To print the lines which has the third word as say "12345" and your log file delimiter is a space (which is generally the case unless you are logging as xml or json), then you could use

cut -d " " -f3 <YOUR LOG FILE NAME> | grep "12345"

where
	-d    " " says that your delimiter is a space
	-f3	  says that you interested in the 3rd delimiter tokenized string

Moving target

What if you wanted to filter lines on a live log file. Just add some "tail" to it and you are good to go

tail -f <YOUR LOG FILE NAME> | grep "ERROR"

Too bad, this does not take care of rolling files.

Less is more

Your production box threw some exception and you are expected to check it out. So, you know that the exception logs exist somewhere towards the end of the file. How do you search for the exception?

Open the file

less <YOUR LOG FILE NAME>

(Don't get me wrong, I love the vi editor but vi tries to load the entire file in its buffer and for large files like our typical log files, performance suffers)

So, yeah less <YOUR FILENAME> will open the file and keep you at the first page. If you need to go to the last page, press

Shift + G

Now, you are on the last line.

Sift though the last pages by pressing

b    (as in back)

and

f    (as in forward)

Elementary, Dr.Watson

So, if you would like to search for "Exception", just type

/Exception

You would get a

"Pattern not found" error

because the default search direction is forward.

To search backward, press

Shift + N

Keep pressing Shift + N to repeat search upwards

If you went past the search string upwards and would like to continue search downwards, then just press

n

and keep pressing n to repeat search downwards

Case-insensitive less search

If you like to make your search keywords insensitive, just type

-i

(remember -i toggles the case insensitivity)

Better to be a head of a dog

tail -f <YOUR LOG FILE> is awesome but if you are analysing the log files inside the less buffer and would like to do a tail of your file, all you need to do is to press

Shift + F

Well, that's all it is for now. I am sure there are many more awesome tricks that you can do in a shell which I don't know or didn't mention here. Please share your favourite trick so that I can do some serious show off to my friends.

PS : There is a way to open the file at the last line with less instead of opening the file at the first page and pressing Shift+G. Can't remember it and can't find it too.