Monitoring System Log Information

I just love Linux. I love each and every aspect of this great operating system. But one fine day, Linux crashed. And not only once, this awesome OS crashed thrice in one single day.
I called upon the services of a good friend and a self proclaimed geek.
He looked into something supposedly called the system log, and soon enough he found the problem and fixed it. Now my favourite OS is working as it should.

But this set me thinking. What was this enigma my dear friend had called 'The System Log'.
What was this awesome thing that helped Linux maintain its favourite position in my eyes.
That was when I decided I had to learn everything I could about system logs
and that was when i decided to share this knowledge with the rest of the world.

SO WHAT IS A SYSTEM LOG?

As it comes out, a system log is a bunch of text files maintained by (almost) all operating systems where they record everything that is taking place inside the OS. You boot your PC, it will be recorded in the log. You insert a pen drive, it will be recorded in the log. You start utorrent, it will be recorded in the log. In short, there is no escaping this monster.

When your systems are running smoothly, you should take some time out to learn and understand the contents of various log files, which will help you when there is a crisis and you have to look though the log files to identify the issue.

UBUNTU

Please note that Ubuntu(Debian based Linux) will be used in the following examples.

For a first look at a log file, fire up terminal and type-
# tail -f /var/log/kern.log

By default, the tail command displays the last 10 lines in a file (kern.log here). You can also specify the number of lines(from the end) that you desire by using tail - eg:tail -50 for 50 lines
the -f option is used to print every new line updated in the file, after the commend has been given. If -f is used, press ctr+c to exit the 'follow' mode

Other options for looking at log files are

# less /var/log/messages
# vi /var/log/messages
# more -f /var/log/messages

OTHER LOG FILES

Now that we are comfortable with accessing a log file, lets have a look at some important log files in Linux.

/var/log/messages – Contains global system messages, including the messages that are logged during system start up. There are several things that are logged in /var/log/messages including mail, cron, daemon, kern, auth, etc.
/var/log/dmesg – Contains kernel ring buffer information. When the system boots up, it prints number of messages on the screen that displays information about the hardware devices that the kernel detects during boot process. These messages are available in kernel ring buffer and whenever the new message comes the old message gets overwritten.
/var/log/auth.log – Contains system authorization information, including user logins.
/var/log/boot.log – Contains information that are logged when the system boots
/var/log/daemon.log – Contains information logged by the various background daemons that runs on the system
/var/log/dpkg.log – Contains information that are logged when a package is installed or removed
/var/log/kern.log – Contains information logged by the kernel. Helpful for you to troubleshoot a custom-built kernel.
/var/log/lastlog – Displays the recent login information for all the users. This is not an ASCII file. You should use lastlog command to view the content of this file.
/var/log/maillog or /var/log/mail.log – Contains the log information from the mail server that is running on the system. For example, sendmail logs information about all the sent items to this file
/var/log/user.log – Contains information about all user level logs
/var/log/Xorg.x.log – Log messages from the X
/var/log/alternatives.log – Information by the update-alternatives are logged into this log file. On Ubuntu, update-alternatives maintains symbolic links determining default commands.
/var/log/btmp – This file contains information about failed login attempts. Use the last command to view the btmp file. For example, “last -f /var/log/btmp | more”
/var/log/cups – All printer and printing related log messages
/var/log/anaconda.log – When you install Linux, all installation related messages are stored in this log file
/var/log/yum.log – Contains information that are logged when a package is installed using yum
/var/log/cron – Whenever cron daemon (or anacron) starts a cron job, it logs the information about the cron job in this file
/var/log/secure – Contains information related to authentication and authorization privileges. For example, sshd logs all the messages here, including unsuccessful login.
/var/log/wtmp or /var/log/utmp – Contains login records. Using wtmp you can find out who is logged into the system. who command uses this file to display the information.
/var/log/faillog – Contains user failed login attempts. Use faillog command to display the content of this file.

Apart from the above log files, /var/log directory may also contain the following sub-directories depending on the application that is running on your system.

/var/log/httpd/ (or) /var/log/apache2 – Contains the apache web server access_log and error_log
/var/log/lighttpd/ – Contains light HTTPD access_log and error_log
/var/log/conman/ – Log files for ConMan client. conman connects remote consoles that are managed by conmand daemon.
/var/log/mail/ – This subdirectory contains additional logs from your mail server. For example, sendmail stores the collected mail statistics in /var/log/mail/statistics file
/var/log/prelink/ – prelink program modifies shared libraries and linked binaries to speed up the startup process. /var/log/prelink/prelink.log contains the information about the .so file that was modified by the prelink.
/var/log/audit/ – Contains logs information stored by the Linux audit daemon (auditd).
/var/log/setroubleshoot/ – SELinux uses setroubleshootd (SE Trouble Shoot Daemon) to notify about issues in the security context of files, and logs those information in this log file.
/var/log/samba/ – Contains log information stored by samba, which is used to connect Windows to Linux.
/var/log/sa/ – Contains the daily sar files that are collected by the sysstat package.
/var/log/sssd/ – Use by system security services daemon that manage access to remote directories and authentication mechanisms.



-http://www.thegeekstuff.com/2011/08/linux-var-log-files/

SOME MORE DETAILS

But how does log work
whenever the OS does any task, it stores a description of the task in the log file
for example-
initially the log file kern.log looks like



after inserting a pendrive


and after removing the pendrive

MULTIPLE LOG FILES AT ONCE

Linux also allows us to access multiple dynamically updated log files in the same terminal.

METHOD-I
the latest version of the tail command allows multiple -f arguments, as shown below
$ tail -f /var/log/syslog -f /var/log/kern.log


The above will display file name(syslog, kern.log) as the first line each time, and then shows the newly grown lines. If you don’t want this to clutter the logs, you can use the next method(my favourite)

METHOD-II
Download multitail as

$ apt-get install multitail

after which you can use the following command
$ multitail /var/log/syslog /var/log/auth.log



Multitail utility has a lot of additional features as explained in the mutitail home page.
For example, you can scroll back in a log file, display log files in colors and search inside log files while using multitail.

-http://www.vanheusden.com/multitail/features.html

ZIPPED LOG FILES

Some of the log files may be zipped, and thus unreadable because of being encoded. To unzip the file, use the following command
$ gunzip a.gz to unzip a.gz
eg-

(note that wtmp.1 is coloured in the first ls, while it is uncoloured in the second one)
also note that you can use
$ gzip a.txt so as to zip a file a.txt

PARSING THROUGH LOG FILES

We can even use parsing so as to extract useful information from the gigantic heaps of data that are log files

Use the command

cat logfilename | grep "keyword"

substituting the name of the log file in logfilename and the keyword to be searched for in keyword.

The output of is taken as input and is searched in each line for the occurence of the . The matched lines are printed.

A USEFUL SHELL SCRIPT TO PARSE THROUGH LOG FILES

Our very ambition in life is to make your life simpler, while not making our life too complicated. Thus we have created this useful(atleast we think its useful) shell script(bash) which can parse through a few important log files.

Copy all the lines in the file https://docs.google.com/document/d/1s5qvidfSUtLdfyMn3-3pH-BjNnEE_lHcJE-8-VG-qzA/edit?hl=en_US onto your favourite text editor, and save it with a '.sh'(without the quotes) extension.Open this file whenever you need to filter the information from the log files

WINDOWS

Its not as if log files are exclusive to Linux. Other operating systems also keep a a record of every message passed in the system.

In Windows Vista, the log files are stored in C:/Windows/System32/LogFiles and various subdirectories within it. Some of the files are readable text files while some are not and exist in various unknown file extensions like .old and .etl
The six subdirectories are Firewall, HTTPERR, Scm, Srt, WMI, WUDF. The text log files SrtTrail and httperr1 in Srt and HTTPERR are two of the important files. Particular information can be found out by searching for the relevant keyword in the text files.

Windows Utility : Event Viewer

However, even though viewing the log files directly in Windows may seem quite impossible, Windows provides with a very powerful solution to the problem. The messages or logs are recorded as small-sized individual messages and are viewable by the "Event Viewer". This tool allows us to view, filter and also store various logs in different formats.




Under Windows Logs, there are 5 options for various type of logs arranged under Application, Security, Setup, System and Forwarded Events. The Application logs store information about the starting and stopping of any service or application whereas the Setup and System logs journal the setting up and installation of services and every system message respectively.


The filter current log tool is a powerful option and permits us to parse the logs according to many criteria viz time, event level, event ID, user or the computer. It also provides an option to view the XML script for a specific query and even manually edit the script to parse out very specific information.

The only feature that can put you off in this utility is that, unlike ubuntu, there is no feature to view the log files in real time update mode ... which means you have to hit refresh to view new events.
When a new device is introduced
When we connected a pen drive, new logs were viewable after refreshing the window which read similar to:

Driver Management concluded the process to install driver FileRepository\wpdfs.inf_amd64_neutral_fc4ebadff3a40ae4\wpdfs.inf for Device Instance ID WPDBUSENUMROOT\UMB\2&37C186B&0&STORAGE#VOLUME#_??_USBSTOR#DISK&VEN_KINGSTON&PROD_DT_100_G2&REV_1.00#001CC0EC321AFB6187162150&0# with the following status: 0x0.

The log was viewable under the System tab which means that this message was passed between the kernel and device driver.

New logs for various subjects are are categorised respectively with the added notification of 'New events available'.

Arhan Sibal 2011031
Rudresh Pande 2011093