Filter lines in log file with ERROR
A couple of days ago, a friend of mine was interested to know how to filter a log file (in my case a log4j log file) for ERRORs alone.
An hour of tricks sharing followed and here is the gist of the conversation that you will be interested in.
Find me wherever I am
Listing all the lines in the log file which has occurrences of ERROR is as simple as executing the following command
grep "ERROR" <YOUR LOG FILE NAME>
When there is a beginning
And if you are interested in the lines which start with ERROR, then it just involves the usage of the regular expression for "begin"
grep "^ERROR" <YOUR LOG FILE NAME>
...there's an end
A similar trick applies if you are interested in searching lines which has occurrences of "Exception" at the end of the line (specially in case of Java stack traces), then you would use
grep "Exception$" <YOUR LOG FILE NAME>
Don't be so sensitive
Good news is that you need not be case sensitive to grep with regard to search string. You could just use
grep -i "exception$" <YOUR LOG FILE NAME>
(i as in insensitive)
That said, imagine you are printing the username or a unique trace id of your request in your logs and you would want to filter your log based on that. And suppose this username/trace id is located as the 3rd word in every line. To print the lines which has the third word as say "12345" and your log file delimiter is a space (which is generally the case unless you are logging as xml or json), then you could use
cut -d " " -f3 <YOUR LOG FILE NAME> | grep "12345"
-d " " says that your delimiter is a space
-f3 says that you interested in the 3rd delimiter tokenized string
What if you wanted to filter lines on a live log file. Just add some "tail" to it and you are good to go
tail -f <YOUR LOG FILE NAME> | grep "ERROR"
Too bad, this does not take care of rolling files.
Less is more
Your production box threw some exception and you are expected to check it out. So, you know that the exception logs exist somewhere towards the end of the file. How do you search for the exception?
Open the file
less <YOUR LOG FILE NAME>
(Don't get me wrong, I love the
vi editor but
vi tries to load the entire file in its buffer and for large files like our typical log files, performance suffers)
less <YOUR FILENAME> will open the file and keep you at the first page. If you need to go to the last page, press
Shift + G
Now, you are on the last line.
Sift though the last pages by pressing
b (as in back)
f (as in forward)
So, if you would like to search for "Exception", just type
You would get a
"Pattern not found" error
because the default search direction is forward.
To search backward, press
Shift + N
Shift + N to repeat search upwards
If you went past the search string upwards and would like to continue search downwards, then just press
and keep pressing
n to repeat search downwards
Case-insensitive less search
If you like to make your search keywords insensitive, just type
-i toggles the case insensitivity)
Better to be a head of a dog
tail -f <YOUR LOG FILE> is awesome but if you are analysing the log files inside the
less buffer and would like to do a
tail of your file, all you need to do is to press
Shift + F
Well, that's all it is for now. I am sure there are many more awesome tricks that you can do in a shell which I don't know or didn't mention here. Please share your favourite trick so that I can do some serious show off to my friends.
PS : There is a way to open the file at the last line with
less instead of opening the file at the first page and pressing
Shift+G. Can't remember it and can't find it too.