Remote Logging of System Log Files
One of the nice features of the linux operating system (and most all unix based systems in general) is they have a system logging facility which allows not only logging system information locally but also remotely with very few changes!
Here is a condensed version of the syslog configuration file info that explains “how” you configure the zonecd to record the information to a remote computer:
SYSLOG.CONF(5) Linux System Administration SYSLOG.CONF(5)
NAMEsyslog.conf – syslogd(8) configuration file DESCRIPTIONThe syslog.conf file is the main configuration file for the syslogd(8) which logs system messages on *nix systems. This file specifies rules for logging. For special features see the sysklogd(8) manpage. Every rule consists of two fields, a selector field and an action field. These two fields are separated by one or more spaces or tabs. The selector field specifies a pattern of facilities and priorities belonging to the specified action. Lines starting with a hash mark (“#”) and empty lines are ignored. This release of syslogd is able to understand an extended syntax. One rule can be divided into several lines if the leading line is terminated with an backslash (“\”). SELECTORSThe selector field itself again consists of two parts, a facility and a priority, separated by a period (“.”). Both parts are case insensitive and can also be specified as decimal numbers, but don’t do that, you have been warned. Both facilities and priorities are described in syslog(3). The names mentioned below correspond to the similar LOG_-values in /usr/include/syslog.h. The facility is one of the following keywords: auth, authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7. The keyword security should not be used anymore and mark is only for internal use and therefore should not be used in applications. Anyway, you may want to specify and redirect these messages here. The facility specifies the subsystem that produced the message, i.e. all mail programs log with the mail facility (LOG_MAIL) if they log using syslog. The priority is one of the following keywords, in ascending order: The keywords: The priority defines the severity of the message The behavior of the original BSD syslogd is that all messages of the specified priority and higher are logged according to the given action. This syslogd(8) behaves the same, but has some extensions. In addition to the above mentioned names the syslogd(8) understands the following extensions: An asterisk (“*”) stands for all facilities or all priorities, depending on where it is used (before or after the period). The keyword none stands for no priority of the given facility. You can specify multiple facilities with the same priority pattern in one statement using the comma (“,”) operator. You may specify as many facilities as you want. Remember that only the facility part from such a statement is taken, a priority part would be skipped. Multiple selectors may be specified for a single action using the semicolon (“;”) separator. Remember that each selector in the selector field is capable to overwrite the preceding ones. Using this behavior you can exclude some priorities from the pattern. This syslogd(8) has a syntax extension to the original BSD source, that makes its use more intuitively. You may precede every priority with an equation sign (“=”) to specify only this single priority and not any of the above. You may also (both is valid, too) precede the priority with an exclamation mark (“!”) to ignore all that priorities, either exact this one or this and any higher priority. If you use both extensions than the exclamation mark must occur before the equation sign, just use it intuitively. ACTIONSThe action field of a rule describes the abstract term “logfile”. A ”logfile” need not to be a real file, btw. The syslogd(8) provides the following actions. Regular File You may prefix each entry with the minus “-” sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner. Named Pipes Terminal and Console Remote Machine Using this feature you’re able to control all syslog messages on one host, if all other machines will log remotely to that. This tears down administration needs. List of Users Everyone logged on EXAMPLESHere are some example, partially taken from a real existing site and configuration. Hopefully they rub out all questions to the configuration, if not, drop me (Joey) a line. # Store critical stuff in critical This will store all messages with the priority crit in the file /var/adm/critical, except for any kernel message. # Kernel messages are first, stored in the kernel The first rule direct any message that has the kernel facility to the file /var/adm/kernel. The second statement directs all kernel messages of the priority crit and higher to the remote host finlandia. This is useful, because if the host crashes and the disks get irreparable errors you might not be able to read the stored messages. If they’re on a remote host, too, you still can try to find out the reason for the crash. The third rule directs these messages to the actual console, so the person who works on the machine will get them, too. The fourth line tells the syslogd to save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info. Everything from err and higher is excluded. # The tcp wrapper loggs with mail.info, we display This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as it’s default. # Store all mail concerning stuff in a file This pattern matches all messages that come with the mail facility, except for the info priority. These will be stored in the file /var/adm/mail. # Log all mail.info and news.info messages to info This will extract all messages that come either with mail.info or with news.info and store them in the file /var/adm/info. # Log info and notice messages to messages file This lets the syslogd log all messages that come with either the info or the notice priority into the file /var/log/messages, except for all messages that use the mail facility. # Log info messages to messages file This statement causes the syslogd to log all messages that come with the info priority to the file /var/log/messages. But any message coming either with the mail or the news facility will not be stored. # Emergency messages will be displayed using wall This rule tells the syslogd to write all emergency messages to all currently logged in users. This is the wall action. # Messages of the priority alert will be directed This rule directs all messages with a priority of alert or higher to the terminals of the operator, i.e. of the users “root” and “joey” if they’re logged in. *.* @finlandia This rule would redirect all messages to a remote host called finlan- (NOTE): You can also use the IP address of the remote machine in dot notation (xxx.xxx.xxx.xxx) in place of the machine name – this way you can specify a syslog server machine that does NOT have a FQDN [Fully Qualified Domain Name) assigned to it in DNS. CONFIGURATION FILE SYNTAX DIFFERENCESSyslogd uses a slightly different syntax for its configuration file than the original BSD sources. Originally all messages of a specific priority and above were forwarded to the log file. The modifiers ”=”, “!” and “-” were added to make the syslogd more flexible and to use it in a more intuitive manner. The original BSD syslogd doesn’t understand spaces as separators between the selector and the action field. FILES/etc/syslog.conf BUGSThe effects of multiple selectors are sometimes not intuitive. For example “mail.crit,*.err” will select “mail” facility messages at the level of “err” or higher, not at the level of “crit” or higher. SEE ALSOsysklogd(8), klogd(8), logger(1), syslog(2), syslog(3) AUTHORSThe syslogd is taken from BSD sources, Greg Wettstein (greg@wind.enjellic.com) performed the port to Linux, Martin Schulze joey@linux.de) made some bugfixes and added some new features. |
Now that you are totally confused how do we use this information to our advantage??
We made a MINOR change to the syslog configuration file that will send ALL the syslog logging information to a remote syslog machine!
Here are the steps needed to configure the zonecd machine so it will send all of the systems logs to a remote syslog host machine:
We create a script file that will reside in the /mnt/floppy/zonecd directory which will write the changes needed to the syslog.conf file in the /etc directory at the correct time then restart the syslogd daemon (program) so the changes take effect.
The following steps are performed either in a GUI terminal window or on the CLI (command line) if you are running in less-x or nox mode:
Perform the following commands to create a file called “syslog.sh” in the proper location in the zonecd machine:
cd /mnt/floppy/zonecd
nano syslog.sh
Now – type in the following:
| #!/bin/sh # # update syslog to send log info to a remote syslog host # replace the xxxxxxxxxxxx with the FQDN of the remote syslog host # If you dont have a FQDN for the remote host (on a private network so no DNS Name#optional depending on if the remote logging host has a valid DNS name or not. # If NOT then perform the following to put a “created” name in the hosts file # so the syslog daemon can “look it up”… # create a name to call the server and then you will need to add the following: # xxx.xxx.xxx.xxx is the IP address of the remote syslog server echo ‘xxx.xxx.xxx.xxx <Created Name>’ >> /etc/hosts echo ‘# send log file info to remote syslog host’ >> /etc/syslog.conf # make sure there is a newline character at the end of the syslog.conf file echo >> /etc/syslog.conf # restart the syslog daemon to implement the logging changes /sbin/sysklogd -r # we are done… |
Type:
Ctrl-x then answer Y to save the file.
(Ctrl-x is performed by pressing and holding the Ctrl-key then pressing the x-key on the keyboard).
You should now have a file in the /mnt/floppy/zonecd directery called:
syslog.sh
You now have to make the file executable so the system can run the file’s contents when the system is booted up (or reboots):
Type: chmod +x syslog.sh
At this point we have an executable file that will configure and restart the syslog facility – but we do not have anything in place to actually cause the file to be executed!
We DO have a way to execute script files during the bootup process – the zonecd system recognizes two distinct executable files during the bootup process where you can have executable content…. the first file is called:
pre.sh
and is executed during the beginning part of the PublicIP bootup process, the second is called:
init.sh
and executes towards the end of the PublicIP bootup process.
We are going to use the pre.sh script file as we want the syslog facility to be sending the PublicIP bootup info to our syslog server when it starts up. This will allow more information for trouble-shooting problems than you would get if you relied on information obtained AFTER the PublicIP system had finished it’s bootup process.
Perform the following to create the pre.sh script file:
(NOTE: If you already have a pre.sh script file on the zonecd system you only need to add the steps after the file creation)
nano /mnt/floppy/zonecd/pre.sh
Type the following:
| #!/bin/sh # # execute the syslog.sh script to allow remote syslog recording/mnt/floppy/zonecd/syslog.sh # we are done here |
Now – save the script file:
type: Ctrl-x and answer Y to save the file.
This file also has to be setup as an executable file so type:
chmod +x /mnt/floppy/zonecd/pre.sh
That’s it – the next time you reboot the system the logs will be sent to a remote syslog host you specified.
Now – What to do with the info – it is useless unless you have something to capture the log info and examine it!
Check on a program called “Splunk” – it is a log file analyzer application that runs on Linux and can be used for the remote syslog server – it will allow you to do all sorts of captured logfile analysis in terms of looking at the log file information in an ordered method.
Gary McKinney
EWCLLC