Forwarding IIS Logs to a central File

Friday, April 2nd, 2004

Forwarding IIS Logs to a central File

Created 2004-04-02 by Timm Herget and Rainer Gerhards.

I would like to centralize IIS log files to a central log server. The files on that central server should be in the exact same format they are on the IIS machines.

This can be done with MonitorWare Agent 2.0 and above. Let’s look into the theory first: If you would like to forward IIS log files AND have them in the same format at the receiving machine, you need to make some special settings.

First of all, please note that the file monitor, when set to "W3C log files", is optimized to extract the properties from each log line, not to forward the log literally. If you would like to forward them literally, you need to make sure that the format is set to "Standard", which will disable all W3C-log specific handling (that would otherwise disturb the result). The syslog tag is not needed here, so it should be totally removed.

We must ensure that the send syslog action does not alter this message content. As such, we must make sure that the "Add Syslog Source when Forwarding" setting is NOT activated.

Unfortunately, that will not eliminate the tag as such from the syslog message, but we can handle this with the property replacer. As of RFC 3164, the syslog tag will be present in the so-generated message. In fact, the message will be ": <ORIGINAL line W3C>" with <ORIGINAL line W3C> literally being the line taken from the W3C log. Effectively, we end up with two extra characters (": ") at the beginning of the line. Thankfully, we can eliminate these with the property replacer (it is capable of providing substrings of event properties). The message is in the "msg" property. So "%msg:3%" is everything from the third character position up until the end of the line (end position is not specified and so "end of line" is the default). To use the property replacer, we must just the "Write to File" action with "Custom" file format. Then, we can enter an arbriatary string that shall be written to the file. In our case, we use "%msg:3%%$CRLF%": this instructs the write to file action to first write the original file line and then a Windows newline sequence. The later is needed because it was stripped out by the file monitor.

This looks in the dialogs as follows:

1. Sender : Forward Via Syslog Settings

The "Add Syslog Source when …"-Checkbox MUST be unchecked.

Figure 1: Forward Syslog Action Settings

2. Sender : File Monitor Service Settings

Please note that the "Syslog Tag Value" Field MUST be empty (not even a space in it).

Figure 2: File Monitor Service Settings

3. Recipient: Syslog Listener Settings

Please note that the "Enable RFC 3164 Parsing" MUST be checked

Figure 3: Syslog Listener Service Settings

4. Recipient: Write to File Action Settings

The "File Path Name" Directory must be available, MonitorWare Agent will not create it if its not present.

The "File Format" MUST be set to "Custom". The following custom line format MUST be used:

%msg:3%%$CRLF%

Figure 4: Write to File Action Settings

With the above settings the recipient MonitorWare Agent  will successfully generate exact the same logfiles as the original ones are.

Sample Configurations

We have created some registry files for both the sender and the recipient server. If you download them, simply import them into the registry on the machine in question (if you system is a default-install, double-clicking the file is sufficient to do this). Be sure that the MonitorWare Agent client is closed while you do this. Please note that the sample configurations MUST be customized in order to make them work for you.

Sample configuration for MonitorWare Agent 2.0

Please note: samples may not work with versions other than the one specified in the download link!

How can I forward IIS logs to a syslog deamon?

Friday, October 4th, 2002

How can I forward IIS logs to a syslog deamon?

Created on 2002-10-04 by Rainer Gerhards.

MonitorWare Agent can forward Microsoft Internet Information Server (IIS) log files to any syslog deamon (or syslo server, if you like). Fortunately, IIS stores web log files as plain text files in the file system. Even better, other processes are allowed to read these files while IIS adds information to them. This enables MonitorWare Agent to forward them in near real-time.

MonitorWare Agent’s file monitor is optimized to pick up application log files. This includes IIS log files. Specific logic enables it to gather only the valid part of the currently being written log file (IIS writes files in 64K increments and there is garbage after the valid log data lines). Special replacement characters inside the file name allow to handle changing file names, so monitoring even works while rolling over to new names.

To activate log forwarding, create one file monitor per IIS log file to monitor. Be sure to use the proper replacement characters if IIS modifies the log file name (by default, it includes the day of month). Details on them can be found in the manual. Then be sure to send all file lines to a rule base that has syslog forwarding enabled. There is a sample in the Step-By-Step Guides inside the manual.

IIS log file data is like any other event data in MonitorWare Agent. So it can not only be forwarded by syslog but also be filtered, acted on, alerts generated and so on. Another possible approach is to generate alerts if specific attack patterns show up in the logs. As long as the pattern is known and can be seen in the log file line, this can easily be configured.

Just a reminder: besides IIS, all other text logs can be processed. Prominent examples include the DHCP log or database message log files.