Author: adisconteam

Database Logging with MSSQL in MonitorWare Agent 4.0

Step-By-Step Guides

Article updated 2006-06-19 by Timm Herget.

Database Logging with MSSQL in MonitorWare Agent 4.0

This guide helps you to add database logging to any of your services available in MonitorWare Agent 4.0.

Microsoft SQL Enterprise Manager

1. To create a new Database, open up the Microsoft SQL Enterprise Manager.

2. Right-click on “Databases” and select “New Database”.

3. Select a Database Name there and click “OK”.

ODBC Data Source Administrator

After you created the new Database, go to the Control Panel -> Administrative Tools and open up “Data Sources (ODBC)”. The following Window will appear:

4. Click on “System DSN” and then “Add…”.

5. Select “SQL Server” as Driver from the List and click “Finish”.

6. Choose a Datasource Name, Description and select the Server where the Database is. In our example we use “localhost”. Click on “Next”.

7. Select “SQL Server Authentication” and type in your MSSQL Login ID and Password. Click on “Next”.

8. Select “Change the default Database to:” and choose your new created Database, in our example we use “MyMWDB”. Click on “Next”.

9. Leave all at default settings and click “Finish”, a test Window will appear:

10. Click on “Test Data Source”, normally the following Window should be displayed:

11. If not, go back and check your Settings, if yes, Click “OK” and exit the System-DSN Wizard.

MonitorWare Agent 4.0

12. To define a new rule set, right click “RuleSets”. A pop up menu as shown below appears. Select “Add Rule Set” from this menu.

13. Then, a wizard starts. Change the name of the rule set to whatever name you like. We will use “Database Logging” in this example. The screen looks as follows:

14. Click “Next”. A new wizard page appears:

15. Select only Database Logging. Do not select any other options for this sample. Also, leave the “Create a Rule for each of the following actions” setting selected. Click “Next”. You will see a confirmation page. Click “Finish” to create the rule set.

16. The wizard closes and the client shows a newly created rule set.

17. As you can see, the new Rule Set “Database Logging” is present. Please expand it in the tree view until the action level of the “Database Logging” Rule and select the “Database Logging” action to configure.
You will see the following Window now:

18. Type in your DSN, User-ID and Password now and press “Save”.

19. Click on the “Create Database” Button and you are shown a pop up screen as shown below:

Here you can specify the required fields and click on create button to get it all done.

How To setup Windows centralized Monitoring

How To setup Windows centralized Monitoring

Article created 2006-02-13 by Timm Herget
Article updated 2006-06-19 by Timm Herget.

Please Note: This article is valid for EventReporter 8.x and lower, WinSyslog 7.x and lower and MonitorWare Agent 4.x and lower in addition to MonitorWare Console 2.1 !

Windows NT/2000/XP/2003 systems monitoring is really important for all small to large sized environments. MonitorWare line of products helps to accomplish this important task. This article is to help you establish a small setup to monitor your Windows NT/2000/XP and 2003 systems.

This article is strictly task focused. It does not describe why the systems should be monitored nor does it provide any further background. Please see the respective backgrounders or each of the products documentation on this. This article is a step-by-step description of what you need to do in order to centrally monitor your Windows NT/2000/XP and 2003 systems.

Centralized Event Reports

In this step-by-step guide, we want to monitor the windows eventlog on all of our client machines (which can be done either with EventReporter or MonitorWare Agent) and then forward the logfiles to a central logserver which writes the data into a database (can be done with WinSyslog or MonitorWare Agent). After this, MonitorWare Console should read the data from this database to automatically generate event summaries for the monitored servers and other devices.

This guide focuses on a typical small to medium business topography with a single geographical location and five windows clients and a central hub server. All systems are well connected via a local ethernet. Event reports from all machines should be stored in a database. The administrator shall receive daily consolidated event reports.

What you need

In this guide, I am focusing on building a solution with Adiscon’s EventReporter, WinSyslog and MonitorWare Console. (Please note that you can use and configure MonitorWare Agent in the same way like either WinSyslog or EventReporter because it is our main product which has all the features of the other two products too. Please also see our article on which product to choose when in doubt which one is right.)
This combination allows you to centralize all your event logs and reports on them. Free 30 day trial versions are available at the respective product sites (links below), so you can try the system without the need to buy anything. You need to run the following products:

  • One EventReporter (alternative: MWAgent) for each system that is to be monitored. In our scenario, this means 6 copies, one for each client and one for the central hub server, if you want to monitor the hub server as well.
  • One WinSyslog (alternative: MWAgent) to receive and store event reports from the EventReporter (alternative: MWAgent) monitoring agents.
  • One MonitorWare Console to automatically generate consolidated reports based on the gathered log data. MonitorWare Console is a very comprehensive tool that helps you to carry out sophisticated analysis of your system. For more information about MonitorWare Console, please refer to its manual.

Notes:

  • To deliver MonitorWare Console reports, you need a local web server (for example Microsoft’s IIS or Apache) and a mail server capable of talking SMTP (most modern servers support this)
  • You need administrative privileges on each of the machines. This is required both for installation and configuration. Make sure you log on with a sufficiently privileged user account.

Step 1 – Download Software

You should check the web sites for new versions if you downloaded your copies a while ago as security and monitoring is a short lived business, and new product versions can appear quickly. Please visit www.eventreporter.com/en/download, eventually www.mwagent.com/download, www.winsyslog.com/en/download and www.mwconsole.com/en/download/ to download the latest versions of EventReporter, MWAgent, WinSyslog and MonitorWare Console.

Step 2 – Installing WinSyslog/MWAgent

Identify the system; WinSyslog or MWAgent (and probably MonitorWare Console) should run on. Take a note of its IP address or host name. You’ll need this value when configuring the EventReporter clients. For our example, I assume this system has an IP address of 192.168.0.1.

Run the WinSyslog/MWAgent setup with default parameters. When setup has finished, it automatically is configured to operate as a simple Syslog server. However, it does not yet use a database as we need it to. We’ll later set it up to write data into the database.

Step 3 – Install EventReporter/MWAgent

Run the EventReporter/MWAgent setup program on all systems that should be monitored. This means you need to run it on all five clients and the central hub server (as mentioned above that it is also to be monitored).

For larger installations (with many more servers) there are ways to set it up in a simpler fashion, but in a scenario like ours, it is faster to install it on each machine manually. You can install it with the default settings. When setup has finished, the program automatically is configured to operate as a simple event reporter. However, it does not yet create the log in our database we need. So we will go ahead and change this on each of the machines or by launching it on one machine and remotely connecting to the others. It is our choice. In this sample, I use the EventReporter/MWAgent on each machine (it is easier to follow).

Step 4 – Create a RuleSet for sending via SETP

The steps to configure the EventReporter/MWAgent on each machine are as follows (repeat this on each of the 5 client machines). This step needs not to be done on the central hub server!:

1. Start EventReporter/MWAgent.

2. Select your language – in this example, I use English, so it might be a good idea to choose English even if that is not your preference. You can change it any time later, but using English makes it much easier to follow this guide here.

3. Then define a new rule set, right click “Rules”. A pop up menu will appear. Select “Add Rule Set” from this menu. On screen, it looks as follows:

4. Then, a wizard starts. Change the name of the rule to whatever name you like. We will use “Sending SETP” in this example and click on Finish. The screen looks as follows:

Now rightclick on the newly created RuleSet and select “Rules”, then “Add Rule” in the upcoming DropDownMenu. The Screen should look as follows:

Expand the new Rule and rightclick on “Actions”, then select “Add Action” and “Send SETP”:

5. After above steps the Action Wizard will come up. Change the name of the rule to whatever name you like. In our case we will name the Action “Send SETP 1”. Click on Finish.

6. Now, type the IP address or host name of our central hub server in the “Servername” field of the newly created Send SETP Action configuration:

7. Make sure you press the “Save” button – otherwise your changes will not be applied.

Step 5 – Create a RuleSet for database logging

This step needs only to be done on the central hub server!

1. Start WinSyslog/MWAgent

2. Again, you can select the language to use. And again, I suggest using english, as this makes the guide easier to follow.

3. Then define a new rule set, right click “Rules”. A pop up menu will appear. Select “Add Rule Set” from this menu. On screen, it looks as follows:

4. Then, a wizard starts. Change the name of the rule to whatever name you like. We will use “Database Logging” in this example. The screen looks as follow:

Click “Next”. A new wizard page appears as appeared in the case of EventReporter/MWAgent. Select only Database Logging here. Do not select any other options for this example. Also, leave the “Create a Rule for each of the following actions” setting selected. Click “Next”. You will see a confirmation page as showned in the case of EventReporter/MWAgent and there you have to click Finish.

5. After above steps you will see that the new Rule Set “Database Logging” is present. Please expand it in the tree view until the action level of the “Database Logging” Rule and select the “Database Logging” action to configure.

6. Now click on the Data Sources (ODBC) button to open the ODBC Data Source Administrator. Then choose the “System DSN” tab and click the “Add” button to add a new System-DSN (Select the Microsoft Access driver like in the screenshot below).

8. In the next step, click the “Select button” and go to the WinSyslog/MWAgent installation directory (Usually C:\program files\*productname*\) and choose the sample database called sample97.mdb. After that name the new DSN with “MyDatabaseDSN” like in the following screenshot and press OK.

9. Now close the ODBC Data Source Administrator and switch back to the WinSyslog/MWAgent Client and insert “MyDatabaseDSN” in the DSN field. Leave all other settings in their default and save the changes.

Step 6 – Create an Event Log Monitor Service

The steps to configure the EventReporter’s/MWAgent’s are as follows. Repeat this step on each of the 5 client machines and the central hub server, if you want to log events from there as well.

Also make sure that there is only one Event Log monitor at a time activated. EventReporter 7.1/MWAgent 3.1 is installed with a default Event Log monitor service. You can use that or create a new one by following these instructions:

1. First, right-click on “Running Services”, then select “Add Service” and the “Event Log Monitor”.

Once you have done so, a new wizard starts.

2. Again, you can use either the default name or any one you like. We will use “My Event Log Monitor” in this sample. Leave the “Use default settings” selected and press “Next”.

3. As we have used the default, the wizard will immediately proceed with step 3, the confirmation page. Press “Finish” to create the service. The wizard completes and returns to the configuration client.

4. Now, you will see the newly created service beneath the “Services” part of the tree view. To check its parameters, select it:

As you can see, the service has been created with the default parameters.

Please note that the “Default RuleSet” has been automatically assigned as the rule set to use. By default, the wizard will always assign the first rule set visible in the tree view to new services. In our case, this is not correct and will be corrected later.

5. Click Advanced Options button in General Options group box. You will be shown a pop up as shown below:

Here check Use Legacy Format and as soon as you check this box, check boxes disabled right now will be enabled and you have to uncheck; Add Username and Syslog Message Numbers.

6. Now we have to make sure that the EventReporter’s/MWAgent’s use the configured “Forward Syslog” Ruleset we created in Step 3. Select that as the rule set to use.

7. Finally, save the changes and  start the EventReporter/MWAgent service. This procedure completes the configuration of the event log forwarder.

EventReporter/MWAgent is not able to dynamically read changed configurations. As such, it needs to be restarted after such changes. In our sample, the service was not yet started, so we simply need to start it. If it already runs, you need to restart it.

With step 5 the client machines configuration has finished. All the next steps are only concerned with the central hub server.

Step 7 – Create a SETP Server Service

The steps to configure the central WinSyslog/MWAgent are as follows (only on central hub server!):

1. First, right click on “Services”, then select “Add Service” and the “SETP Server”.

Once you have done so, a new wizard starts.

2. Again, you can use either the default name or any one you like. We will use “My SETP Server” in this sample. Leave the “Use default settings” selected and press “Next”.

3. As we have used the default, the wizard will immediately proceed with step 3, the confirmation page. Press “Finish” to create the service. The wizard completes and returns to the configuration client.

4. Now, you will see the newly created service beneath the “Services” part of the tree view. To check its parameters, select it:

As you can see, the service has been created with the default parameters.

5. To use the “Database Logging” RuleSet we have created in Step 4, select it as rule set to use.

6. Last, save the change and then restart the WinSyslog/MWAgent service. This procedure completes the configuration of the SETP server.

WinSyslog/MWAgent cannot dynamically read changed configurations. As such, it needs to be restarted after such changes.

Step 8 – Preparing Web Server for MonitorWare Console

MonitorWare Console publishes its reports through the local web server (central hub server). To avoid confusion, we recommend creating a separate directory on the web server for MonitorWare Console. Let us assume that you use Microsoft Internet Information Server and run it with the default configuration. Then, your web pages are stored in the c:\inetpub\wwwroot directory. Create a subdirectory “MonitorWare Console” directly under this directory.

Step 9 – Installing and Configuring MonitorWare Console

Step 10 – Generating Reports with MonitorWare Console Manually

This section explains how the reports can be generated with MonitorWare Console manually. Since “System Status” Report is most comprehensive report that tells a detailed description about the network, in this section I will explain this report only. Please note, that the procedure for generating any report is almost the same.

Step 11 – Scheduling the Generation of Reports with MonitorWare Console

This section explains how the reports can be generated with MonitorWare Console automatically using Job Manager. With Job Manager, you can generate all the reports based on a pre-defined schedule and ask it to either store it in some location on the hard disk or send it to specified recipient via email. Once again, I will explain the scheduling of System Status Report in this section. Please note that, the procedure for scheduling any report is the same.

You are done!

Well, this is all you need to do to configure the basic operations. Once you are comfortable with the basic setup, you can enhance the system with local pre-filtering of event, enhanced logging and alerting (with MonitorWare Agent) and changing report options (with MonitorWare Console).

I hope this article is helpful. If you have any questions or remarks, please do not hesitate to contact me at therget@hq.adiscon.com.

How To setup PIX centralized Monitoring

How To setup PIX centralized Monitoring

Article created 2005-05-17 by Hamid Ali Raja
Last Updated 2006-06-19 by Timm Herget.

Adiscon Products can be used to efficiently analyze PIX traffic as well. This article is strictly task focused. It does not describe why the systems should be monitored nor does it provide any further background. Please see the respective backgrounders or product documentation on this. This article is a step-by-step description of what you need to do in order to centrally monitor your PIX Logs.

Centralized Event Reports

In this step-by-step guide, MonitorWare Agent / WinSyslog is configured to work together with Adiscon’s MonitorWare Console to generate summaries for the traffic passing to and from PIX.

What you need

In this guide, I am focusing on building a solution with Adiscon’s MonitorWare Agent / WinSyslog and MonitorWare Console. This guide will be equally good for you if you want to configure MonitorWare Console with MonitorWare Agent or to configure MonitorWare Console with WinSyslog. The reason is that in this configuration we need to have a Syslog Daemon that will be listening for Syslog messages. Since both MonitorWare Agent and WinSyslog can act as  Syslog Daemons and since the setup procedure for both of them as a Syslog Daemon is exactly the same, this guide can be used for both the cases.

This combination allows you to centralize all your logs and generate reports on them. Free 30 day trial versions are available at the respective product sites (links below), so you can try the system without the need to buy anything.

You need to run the following products:

  • 1 MonitorWare Agent / WinSyslog for the system that will act as the Syslog Daemon.
  • 1 MonitorWare Console to generate consolidated reports based on the gathered log data. This will also be installed on the same machine where you have installed MonitorWare Agent / WinSyslog

  • To deliver MonitorWare Console’s reports, you need a local web server (for example Microsoft’s IIS or Apache) and a mail server capable of talking SMTP (most modern servers support this)

You need administrative privileges on each of the machines. This is required both for installation and configuration. Make sure you log on with a sufficiently privileged user account.

Step 1 – Download Software

You need to download the following software to follow this step by step guide:

1. www.mwagent.com/download or www.winsyslog.com/en/download
2. www.mwconsole.com/en/download

Step 2 – Install MonitorWare Agent / WinSyslog

Run the MonitorWare Agent / Winsyslog setup program on the system that is to act as the central server. Take a note of this server’s IP address or host name. You’ll need this value when configuring PIX to forward the messages to it.

Step 3 – Configure a Syslog Server

The steps to configure the MonitorWare Agent / WinSyslog as a Syslog Server are as follows:

Configuring a Syslog Server

Step 4 – Create a RuleSet for Database Logging

In this section, you will create an action to write the messages that are coming from PIX to a database. Please note that these steps would be exactly the same for both MonitorWare Agent and WinSyslog.

Database Logging Steps

After configuring this RuleSet, make sure that

  • This rule set is associated with the Syslog Server Service that you created in Step 3. You can do this by clicking on the Syslog Server Service that you created in Step 3 on the left hand side and by selecting the name of the rule set that you created in Step 4 in “Rule Set to Use” combo box on the right hand side.
  • The service is running. You can do this by clicking on the Play button at the top of the Client.

Step 5 – Configure PIX

In this step, you will need to configure PIX in such a way so that it sends the messages to the Syslog Server that you created in the above step. You would need to give the IP address or the hostname in PIX

PIX Configuration Steps

Step 6 – Preparing Web Server for MonitorWare Console

MonitorWare Console publishes its reports through the local web server (central hub server).

To avoid confusion, we recommend creating a separate directory on the web server for MonitorWare Console. Let us assume you use Microsoft Internet Information Server and run it in the default configuration. Then, you web pages are stored in the c:\inetpub\wwwroot directory. Create a subdirectory “MonitorWareConsole” directly beneath this directory.

Step 7 – Installing and Configuring MonitorWare Console

MWConsole- Installation and Configuration Steps

Step 8 – Generating PIX Reports with MonitorWare Console Manually

Following are the reports in MonitorWare Console that can be generated for PIX logs.

  • Accessed Web Sites Report
  • Blocked Ports Activity Report
  • Possible Attacks Report
  • Traffic By Hour Report
  • Traffic By Port Report
  • Outbound Traffic By IP
  • Traffic by Target IP
  • Who is Attacking Me Report

This section explains how the PIX reports can be generated with MonitorWare Console manually. In this section I will explain the generation of “Who is Attacking Me” report only. Please note that, the procedure for generating any report is almost the same.
Generating PIX Reports with Console 2.1 Manually

Step 9 – Scheduling the Generation of Reports with MonitorWare Console

This section explains how the reports can be generated with MonitorWare Console automatically using Job Manager. With Job Manager, you can generate all the reports based on a pre-defined schedule and ask it to either store it in some location on the hard disk or send it to specified recipient via email. The following section explains the scheduling of System Status Report. You can use exactly the same method to generate any of the PIX reports that are mentioned above.

Scheduling Reports with Console 2.1

You are done!

Well, this is all you need to do to configure the basic operations. We hope this article is helpful. If you have any questions or remarks, please do not hesitate to contact us at support@adiscon.com

2006-05-09 MonitorWare Agent 4.0 Final (Build Service 4.0.306/Client 4.0.1020)

MonitorWare Agent 4.0 Released

Build-IDs: Service 4.0.306, Client 4.0.1020

New Additions

  • x64 Build

    This is the first Version which introduces MonitorWare Agent on the x64 platform. Major compatibility changes for the x64 platform have been made in the Service core. For details see the changes listed below:
    – ODBC Database Action, fully runs on x64 now. Please note that there are currently very few ODBC drivers for x64 available!
    – Configuration Registry Access, a DWORD Value will now be saved as QWORD into the registry. However the Configuration Client and Win32 Service Build can handle these data type and convert these values automatically into DWORD if needed. The Configuration Client will remain a win32 application. Only the Service has been ported to the x64 platform. Continue reading “2006-05-09 MonitorWare Agent 4.0 Final (Build Service 4.0.306/Client 4.0.1020)”

Forwarding filtered IIS Logfiles

Forwarding filtered IIS Logfiles

Created 2006-04-19 by Timm Herget

Please note: In order to forward the IIS logs you need MWAgent.

Step 1

First, create a new RuleSet, in our sample we named it ForwardSyslog, and bind an ForwardSyslog ,or any other action you want to use for forwarding (e.g. SendEmail), to it. In our sample we now must define the syslog server where we want to forward it to:


Figure 1: Creating the ruleset and its action

Step 2

Then create your Filemonitor and point it to the location of your IIS Logfile which you want to monitor.
(Note: Daily Internet Information Server log files are named “exyymmdd.log”, with yy being the 2 digit year, mm the month and dd the day of month. To generate the same name with file monitor, use the following name “ex%y%m%d.log”.)
Set the Logfile Type to “W3C WebServer Logfile” and assign it to your newly created RuleSet (in our case its ForwardSyslog).


Figure 2: Creating and configuring the filemonitor

Step 3.1

Click on the FilterConditions Tab of your newly created RuleSet, which contains your forwarding action (in our case the ForwardSyslog ruleset). Set the filters as shown on the screen below. In our sample we want to forward every log entry, excepting accesses from client ip (c-ip) 127.0.0.1 so we set the operator to AND and use a Custom Property. Set the Property Name to “p-c-ip”, the compare operation to “does not contain” and the Property Value to “127.0.0.1”. In pseudo code this looks like this:

  • if (IP != “127.0.0.1”)
    {
    forward the logentry
    }
    else
    {
    discard the logentry
    }


Figure 3: Setting up the filterconditions

Step 3.2

Of course you can add as many ip’s to filter out as you want to. Simply change the introducing AND to OR and set those filters.
To do so simply click on the AND and then on “Change Operator”.
This means “If ip-x OR ip-y OR ip-z are not found, do x”:


Figure 4: Advancing filterconditions part one

Step 3.3

If you would specify the filters in that way, that you only want to forward some specific IP’s and not, as we mentioned above,
all excepting some specific then set the initial operator to OR and the filteroption from “does not contain” to “must contain”
and configure your IP filters:


Figure 5: Advancing filterconditions part two

(Note: There are some more special IIS filter options, read more in the online manual.)

How to store custom properties of a log message in a database

How to store custom properties of a log message in a database

Created 2006-03-27 by Timm Herget

This step-by-step guide describes a scenario where WinSyslog receives syslog data from a Fortigate firewall, parses the messages via post processing action and writes the custom parsed properties into a database.

Step 1 – Creating the Syslog Server

First, please create the syslog server service by right clicking on “Running Services” and selecting “Add Service” and “Syslog Server” from the upcoming dropdpwn menu. After creating it, leave the settings default at this time:


Figure1: creating the syslog server service

Step 2 – Creating the RuleSet

Now, right click on “RuleSets” and select “Add RuleSet”, type in a name for the ruleset and click on “Finish”:


Figure2: creating the ruleset

Step 3 – Creating the Rule and its Action

Right click on the newly created RuleSet and select “Add Rule”, again type in a name and click on “Add Rule”. Now right click on “Actions”, select “Add Action” and then “Post-Process Event”:


Figure3: creating the rule and action

Step 4 – The Syslog Data from the Fortigate Firewall

Below you can see a sample logfile of a Fortigate firewall. All properties are seperated by a space. The highlited properties we want to parse and log them to a databse in our step-by-step guide:


Figure4: the sample logfile

Step 5 – Configuring the PostProcess Action

Now we can start configuring the post process action. At first, please click on “Insert” to create the first property we want to configure for parsing. Change the property to “Filler” and the type to “UpTo” from the dropdownlists above. Also insert “date=” as value for this rule. In short this says: Take all the characters up to the begin of “date=” and drop them away, the cursor now stands at the first character of “date=” which is the “d”:


Figure5: first parsing rule

Again, click on “Insert”. Set the property to “Filler”, the type to “Character Match” and the value to “date=”. This will tell the parser to set the cursor to the first character after the string “date=” (Note: Character Match begins “searching to the end of string” from the begin of the current cursor position ONLY, so we need to jump to “date=” first via a UpTo rule (already done one step above)):


Figure6: second parsing rule

As I told above, the cursor of the parser now stands at the first character after “date=” which means we are now able to parse the first value from the logfile to a custom property: The date. To tell the parser to do so, please again click on “Insert”, name the property “u-date” or anything you want and set the type to “Word”. This will parse all characters from where the curser currently is up to the next space he will find. The date from the firewall syslog data is now stored into the property “u-date”.


Figure7: third parsing rule

Insert the next rule, set the property to “Filler”, the type to “UpTo” and the value to “time=”. Again click “Insert”, set the property to “Filler”, the type to “Character Match” and the value to “time=”. These two new rules will take all characters from the current cursor position to the last one of the string “time=” and drop them away. The cursor now stands on the first character after “time=”.


Figure8: fourth parsing rule

Again we have to insert a new rule, name it “u-time”, set the property to “UpTo” and the value to ” device_id=” (Note the space before device_id). This will write all the characters after “time=” until ” device_id=” into the property u-time. We now have the time too.


Figure9: fifth parsing rule

Insert the next two rules, for the first please set the property to “Filler”, the type to “UpTo” and the value to “SN=”. For the second please set the property to “Filler”, the type to “Character Match” and the value to “SN=” too. The cursor now stands on the first character after “SN=”.


Figure10: sixth parsing rule

Now create the new rule named “u-SN” which indicades the SerialNumber property of the syslog message from the firewall, set the type to “Integer” and click on Save. This will parse all Integer characters from the current cursor position, until no more Integer character is found, into “u-SN”.


Figure11: seventh parsing rule

We got 3 out of 4 properties now from the syslog data, so one is still remaining. Here we go. Again create two rules, both as “Filler” and with value “src=”, the first of them of type “UpTo” and the other “Character Match”. The cursor now stands on the first character after “src=”.


Figure12: eighth parsing rule

Now we only have to parse out the Source-IP which we will do via a Word type rule. Name the next Rule “u-source” and set the type to “Word”. This will parse all from the current cursor position until the next space into “u-source”. Now we got all four properties we wanted.


Figure13: last parsing rule

Step 6 – Creating the Database

Create a new Database via i.e. MySQL or MSSQL. In our sample we create it within MySQL and use PHPMyAdmin as DatabaseTool. First create the Database, we named it “step-by-step-dbsample” and then create a new table (“sample_properties”) with the following fieldnames:

  • Date
  • Time
  • SN
  • SourceMachine

The SQL Statements for this two steps should look like these:


Figure14: creating the database

Step 7 – Creating the System DSN

Now we must create a system dsn for our newly created table, so that we can access it via our database logging action later. To do so, please go to “Start” -> “Control Panel” -> “Administrative Tools” -> “Data Sources (ODBC)”. Go to the tab “System DSN” and click “Add”. Select the Driver you need from the shown list, in our sample this is “MySQL ODBC 3.51 Driver”, and click “Finish”. Please then fill in the shown configuration form in that way, that it suits your database and click on “Test”. If the test succeeds you are done, click OK then. If it failed, please re-check your configuration data. Remember the configured “Source Name”.


Figure15: testing system dsn

Step 8 – Configuring the Database Logging Action #1

Now please add a “Write to Database” action to your rule created at the beginning. To do so, expand the treeview of your “Post-Process-Action” rule in our ruleset, right click on “Actions” -> “Add Action” -> “Write to Database”. You will see the newly created Action like shown below:


Figure16: database logging action #1

Step 9 – Configuring the Database Logging Action #2

First, type in your new system dsn name in the “DSN” field, then the suiting username and password. Change the “Table Name” to the name of the table we created in step 6. After this, delete all the configured fields in the data-grid-view. Select them and then click on the “Delete” Button. After these steps are done, it should look like the following screenshot:


Figure17: database logging action #2

Step 10 – Configuring the Database Logging Action #3

Now we have to configure the database logging action in that way, that it writes our custom parsed properties into the fields of the new database/table. Please click on “Insert” first to create a new “row”. Type in the fieldname of our first field in the database/table. In our sample this is “Date”. Set the fieldname to “Date”, the fieldtype to “varchar” and the fieldcontent to “u-date” (where we parsed in the userdefined date before). Do the same now for the “Time” Field. Again click on “Insert”, set the fieldname to “Time”, the fieldtype to “varchar” and the fieldcontent to “u-time”. Repeat these steps for the two remaining fields with fieldtype “varchar” too. Click on “Save”:


Figure18: database logging action #3

Step 11 – You are done

If you now take a look into your Database (in our sample MySQL via PHPMyAdmin) you will see that it worked fine and the message properties were correctly parsed and stored (in our sample i sent 3 of those messages).


Figure19: The Result Please Note: There’s also an Article available which describes how the parsing of logfiles works, you can find it here.

Attachments:

How to store custom properties of a log message in a database

How to store custom properties of a log message in a database

Created 2006-03-27 by Timm Herget

This step-by-step guide describes a scenario where WinSyslog receives syslog data from a Fortigate firewall, parses the messages via post processing action and writes the custom parsed properties into a database.

Step 1 – Creating the Syslog Server

First, please create the syslog server service by right clicking on “Running Services” and selecting “Add Service” and “Syslog Server” from the upcoming dropdpwn menu. After creating it, leave the settings default at this time:


Figure1: creating the syslog server service

Step 2 – Creating the RuleSet

Now, right click on “RuleSets” and select “Add RuleSet”, type in a name for the ruleset and click on “Finish”:


Figure2: creating the ruleset

Step 3 – Creating the Rule and its Action

Right click on the newly created RuleSet and select “Add Rule”, again type in a name and click on “Add Rule”. Now right click on “Actions”, select “Add Action” and then “Post-Process Event”:


Figure3: creating the rule and action

Step 4 – The Syslog Data from the Fortigate Firewall

Below you can see a sample logfile of a Fortigate firewall. All properties are seperated by a space. The highlited properties we want to parse and log them to a databse in our step-by-step guide:


Figure4: the sample logfile

Step 5 – Configuring the PostProcess Action

Now we can start configuring the post process action. At first, please click on “Insert” to create the first property we want to configure for parsing. Change the property to “Filler” and the type to “UpTo” from the dropdownlists above. Also insert “date=” as value for this rule. In short this says: Take all the characters up to the begin of “date=” and drop them away, the cursor now stands at the first character of “date=” which is the “d”:


Figure5: first parsing rule

Again, click on “Insert”. Set the property to “Filler”, the type to “Character Match” and the value to “date=”. This will tell the parser to set the cursor to the first character after the string “date=” (Note: Character Match begins “searching to the end of string” from the begin of the current cursor position ONLY, so we need to jump to “date=” first via a UpTo rule (already done one step above)):


Figure6: second parsing rule

As I told above, the cursor of the parser now stands at the first character after “date=” which means we are now able to parse the first value from the logfile to a custom property: The date. To tell the parser to do so, please again click on “Insert”, name the property “u-date” or anything you want and set the type to “Word”. This will parse all characters from where the curser currently is up to the next space he will find. The date from the firewall syslog data is now stored into the property “u-date”.


Figure7: third parsing rule

Insert the next rule, set the property to “Filler”, the type to “UpTo” and the value to “time=”. Again click “Insert”, set the property to “Filler”, the type to “Character Match” and the value to “time=”. These two new rules will take all characters from the current cursor position to the last one of the string “time=” and drop them away. The cursor now stands on the first character after “time=”.


Figure8: fourth parsing rule

Again we have to insert a new rule, name it “u-time”, set the property to “UpTo” and the value to ” device_id=” (Note the space before device_id). This will write all the characters after “time=” until ” device_id=” into the property u-time. We now have the time too.


Figure9: fifth parsing rule

Insert the next two rules, for the first please set the property to “Filler”, the type to “UpTo” and the value to “SN=”. For the second please set the property to “Filler”, the type to “Character Match” and the value to “SN=” too. The cursor now stands on the first character after “SN=”.


Figure10: sixth parsing rule

Now create the new rule named “u-SN” which indicades the SerialNumber property of the syslog message from the firewall, set the type to “Integer” and click on Save. This will parse all Integer characters from the current cursor position, until no more Integer character is found, into “u-SN”.


Figure11: seventh parsing rule

We got 3 out of 4 properties now from the syslog data, so one is still remaining. Here we go. Again create two rules, both as “Filler” and with value “src=”, the first of them of type “UpTo” and the other “Character Match”. The cursor now stands on the first character after “src=”.


Figure12: eighth parsing rule

Now we only have to parse out the Source-IP which we will do via a Word type rule. Name the next Rule “u-source” and set the type to “Word”. This will parse all from the current cursor position until the next space into “u-source”. Now we got all four properties we wanted.


Figure13: last parsing rule

Step 6 – Creating the Database

Create a new Database via i.e. MySQL or MSSQL. In our sample we create it within MySQL and use PHPMyAdmin as DatabaseTool. First create the Database, we named it “step-by-step-dbsample” and then create a new table (“sample_properties”) with the following fieldnames:

  • Date
  • Time
  • SN
  • SourceMachine

The SQL Statements for this two steps should look like these:


Figure14: creating the database

Step 7 – Creating the System DSN

Now we must create a system dsn for our newly created table, so that we can access it via our database logging action later. To do so, please go to “Start” -> “Control Panel” -> “Administrative Tools” -> “Data Sources (ODBC)”. Go to the tab “System DSN” and click “Add”. Select the Driver you need from the shown list, in our sample this is “MySQL ODBC 3.51 Driver”, and click “Finish”. Please then fill in the shown configuration form in that way, that it suits your database and click on “Test”. If the test succeeds you are done, click OK then. If it failed, please re-check your configuration data. Remember the configured “Source Name”.


Figure15: testing system dsn

Step 8 – Configuring the Database Logging Action #1

Now please add a “Write to Database” action to your rule created at the beginning. To do so, expand the treeview of your “Post-Process-Action” rule in our ruleset, right click on “Actions” -> “Add Action” -> “Write to Database”. You will see the newly created Action like shown below:


Figure16: database logging action #1

Step 9 – Configuring the Database Logging Action #2

First, type in your new system dsn name in the “DSN” field, then the suiting username and password. Change the “Table Name” to the name of the table we created in step 6. After this, delete all the configured fields in the data-grid-view. Select them and then click on the “Delete” Button. After these steps are done, it should look like the following screenshot:


Figure17: database logging action #2

Step 10 – Configuring the Database Logging Action #3

Now we have to configure the database logging action in that way, that it writes our custom parsed properties into the fields of the new database/table. Please click on “Insert” first to create a new “row”. Type in the fieldname of our first field in the database/table. In our sample this is “Date”. Set the fieldname to “Date”, the fieldtype to “varchar” and the fieldcontent to “u-date” (where we parsed in the userdefined date before). Do the same now for the “Time” Field. Again click on “Insert”, set the fieldname to “Time”, the fieldtype to “varchar” and the fieldcontent to “u-time”. Repeat these steps for the two remaining fields with fieldtype “varchar” too. Click on “Save”:


Figure18: database logging action #3

Step 11 – You are done

If you now take a look into your Database (in our sample MySQL via PHPMyAdmin) you will see that it worked fine and the message properties were correctly parsed and stored (in our sample i sent 3 of those messages).


Figure19: The Result Please Note: There’s also an Article available which describes how the parsing of logfiles works, you can find it here.

Attachments:

Parsing log messages

Parsing log messages

Created 2006-03-24 by Michael Meckelein.

This article describes how to parse log message via “Post-Process”. It illustrates the logic behind Post-Process action.

Get relevant information from logs

Log files contain a lot of information. In most cases only a small part of the log message is of actual interest. Extracting relevant information is often difficulty. Due to a variety of different log formats a generic parser covering all formats is not available.

Good examples are firewalls. Cisco PIX and Fortigate firewalls both use syslog for logging.. But the content of their respective log messages are very different. Therefore a method is needed to parse the logs in a generic way. Here Post-Process action of Adiscon’s MonitorWare comes into play.

Tool kit for parsing

Post-Process action provides an editor for creating a log format template. A template consists of as many rules as necessary to parse out the relevant information.

Determine necessary information

In order to parse out information it is vital to know the exact structure of the message. Identifying the position of each relevant item is essential. Assuming for auditing purposes the following items are needed:

Timestamp | Source IP-Address | SyslogTag | MessageID | Username | Status | Additional Information

A sample message looks like:

Mar 29 08:30:00 172.16.0.1 %Access-User: 12345: rule=monitor-user-login user=Bob status=denied msg=User does not exist

In order to extract the information let us examine each item within the message. Splitting the message makes it easier to explain. So here we go.

Pos = Position of the character.
*p  = Points to the position the parser stands after parsing the rule.
Log = Message subdivided into its characters.
Pro = Property. In the term of Adiscon a property is the name of the item which is parsed out.

Note that at beginning of the parse process the parser’s pointer points to the first character. Each parse type starts parsing at the current position of the pointer.

Parsing out a Timestamp

The first identified item is a so called Unix/Timestamp. It has always a length of 15 characters. ‘UNIX/LINUX-like Timestamp’ parse type exactly covers the requirement to parse this item. Therefore insert a rule and select ‘UNIX/LINUX-like Timestamp’ type. This rule parses out the timestamp and moves the pointer to the next character after the timestamp. Name the property ‘u-timestamp’ [1].


Post-Process Editor: Inserted a ‘UNIX/LINUX like timestamp’ rule

Get the IP-Address

Next item is the IP address. Note that after the timestamp follows a space and then the IP address. Therefore insert a ‘Character Match’ rule with a space as value. Select the ‘Filler’ [2] property for this rule. ‘Character Match’ requires a user defined value. This parse type compares the given value with the character at the current position of the message. The character has to be identical with the given value otherwise the parse process will fail. After applying this parse type the parse pointer is moved to the position immediately after the given value. In our sample this is the start position of the IP Address (Pos 17).

After that the address can be obtained. Place in a ‘IP V4 Address’ type. This type parses out a valid IP regardless of its length. No need to take care about the characters. Select ‘Source’ property or name it to whatever you prefer. The parser will automatically move the pointer to the position next to the address.


Note the value of ‘Character Match’ rule is a space.

Obtain the syslogtag

Behind the IP it is a blank followed by a percent sign. The percent indicates that the syslogtag is following. To move the pointer to the syslogtag position once again a ‘Character Match’ rule is necessary. It has to match the space (actual position of the pointer) and the percent sign. This content is not needed therefore assign it to the ‘Filler’ property.

A colon is immediately behind the syslogtag. So all characters between the percent sign and the colon are needed. The ‘UpTo’ type can do this job. Insert an ‘UpTo’ rule. As value enter ‘:’ (without the quotes) and select the syslogtag property. Note that after parsing the pointer stands on the first character of the ‘UpTo’ value.


Important: It points to the colon not to the blank.

Take the MessageID

The next interesting item is the MessageID. Move the pointer to start position of the MessageID part. Again, do this by using a ‘Character Match’ rule. Keep in mind that the pointer points to the colon. Behind the colon is a space and then the MessageID starts. Thus, the value of the rule has to be ‘: ‘.

MessageID consist of numbers only. For numeric parsing the ‘Integer’ parse type exist. This type captures all characters until a non-numeric character appears. The pointer is moved behind the number. Note that numeric values with decimal dots can not be parsed with this type (because they are not integers). This means trying to parse 1.1 results in 1, because the dot is a non-numeric value.

Find the username and status

Looking at the remainder of the message indicates that the username is not immediately after syslogtag. Thankfully though, the username always starts with ‘user=’. Consequently the ‘UpTo’ type can be used to identify the username. To get the start position of the username we have to use ‘UpTo’ together with ‘Character Match’. Remember that ‘UpTo’ points to the first character of the given value. For this reason the ‘Character Match’ rule is necessary.

After locating the start position of the username ‘Word’ parse type can be used. ‘Word’ parses as long as a space sign is found. Enter ‘u-username’ as property.


Notice: After parsing a word the pointer stands on the space behind the parsed word.

The steps to get the status are very similar to the previous one.

The last rule – Additional Information

One item of interest is left. The last part of the message contains additional information. It starts after ‘msg=’. So the combination of ‘UpTo’ and ‘Character Match’ is used to go to the right position. All characters after ‘msg=’ until the end of the message are interesting. For this purpose the ‘Rest of Message’ parse type is available. It stores all characters from the current position until the end of the message. This also means that this rule can only be used once in a template and is always the last rule.


Complete parse template.

What happens if the parser fails?

If a rule does not match processing stops at this point. This means all properties of rules which were processed successfully until the non-matching rule occurs are available.

Let’s assume the fourth rule of the following sample does not match.

The first three rules were processed successfully. Therefore u-timestamp and Source are available. But syslogtag and u-messageid are always empty due to the parser never process this rules.

The Post-Process template which was created in this article is available for download. If you have further question on Post-Process, please contact our support.

[1] Using the “u-” prefix is recommended to differentiate between MonitorWare-defined properties and user defined one. It is not required, but often of great aid. A common trap is that future versions of MonitorWare may use property names that a user has also used. MonitorWare will never use any name starting with “u-“, so the prefix also guards against such a scenario.

[2] Filler is a predefined property which acts as a bin for unwanted characters. Essentially, the data is simply discarded.

Please Note: There’s also a StepByStep Guide available which describes how the PostProcessAction works, you can find it here.

IIS Workflow Described

IIS Workflow Described

By Rainer Gerhards
Article Date: 2003-03-20

Abstract

This paper describes the IIS workflow (aka “order of operations”) as far as the author thinks it is. I have tried hard to make the information as complete and accurate as possible, but obviously it might be wrong as I haven’t coded IIS. All information in this paper is taken from official Microsoft documenation as well as working with IIS in lab.

This paper will become part of a larger IIS-logging focussed paper. In fact, I excerpted it from a later release, because I see increasing intereste in IIS workflow caused by some new malware on the web. So this paper is not fully proof-read and there will most probably be some typos, gramatical errors as well as inconsistencies. Especially if you spot an inconsistency, I would appreicate if you let me know at rgerhards@adiscon.com because I obviously would like to have it as easily understandable as possible.

The above mentioned IIS logging paper can be found at http://www.monitorware.com/common/en/SecurityReference/monitoring-iis-logs.pdf

Additional information and papers may also be available at http://www.monitorware.com/common/en/SecurityReference/.

Please note that this document is correct for IIS up to version 5.1. Microsoft has announced considerable change for IIS 6.0, and the information in this document might not be valid for that version.

IIS order of Operations

Conceptually, IIS has three processing stages:

  1. Decoding the target (web page) of the request
  2. Serving the request
  3. Finishing the request, which includes logging

Before we dig into the details, be sure to fully understand the implications of that overall order of processing: no data is logged by IIS before the request is finished. The reason for this is that IIS log files contain data like the total number bytes send or the processing time required. This data is only available when the request is finished. This also means that when an attacker either crashes IIS or exploits a vulnerability that does not allow it to reach step 3, no log data is written. As a side note, the same is true for Apache.

The problem of missing log data from web servers failing in stages one or two can only be addressed by writing two log lines – one at the begin of the request and one at the end. This is currently not supported by IIS (nor other web servers I know of). Adiscon is developing a tool that allows to do this. Please email info@adiscon.com if you are interested in this and would like to receive a copy as soon as it is available.

Now let us dig into the details of the IIS order of operations. A word of warning first: We have compiled this information from what is published in the Microsoft IIS SDK as well as our own testing. We made any attempt to do do it correctly. However, there is a slight chance that we are failing in one regard or another, so please be careful when basing your work on the information contained in here. Depending on what you try to do, it can be a good idea to re-confirm the information in question with somebody at Microsoft.

First of all, it is important to remember that http request come in via TCP. TCP is a stream-oriented protocol, not a packet-oriented one. This means that any TCP application (like IIS) needs to expect that multiple reads of the TCP stream can be necessary to read a piece of information – even if it is a small one. A practical sample: the URL of the requested page is contained in the very first bytes of the HTTP header. So typically, this information is readily available when IIS receives the first TCP packet and begins processing of it. However, it is possible that someone manually connects to IIS via telnet to port 80. As he then types the HTTP header on the keyboard, each character will be delivered in its own packet. As such, the initial packet will just hold one character and not the complete URL that is typically in there. We need to keep this in mind when talking about the IIS order of operations. Some of the stages described here will typically only be involved once, but there might be situations where multiple executions of these stages code base might be necessary. This is more an implementors issue, but it can be a challenge to log analysis and securing IIS as vulnerabilites, too might be caused by implementors not appropriately handling such situations (this especially applies to ISAPI filters, which require a deep understanding of what IIS does).

Stage 1: Decoding the Request

This initial phase is used to read the HTTP headers and identify the page (or script) to be called by the web server. This phase also involves authentication for non-anonymous access.

IIS reads the TCP stream until sufficiently information has been gathered. “Sufficiently” means at least the actual request type (line one of the HTTP header) as well as the host headers and such. I did not find a clear indication of what “sufficient” is. In my point of view, I think it is the complete HTTP header, but I can not prove this to be 100% exact.

Please note that when “NT Challenge / Response” authentication is used. IIS will even initiate the challenge / response process with the client. This involves sending the challenge and receiving the response, two network packets, as part of the initial phase.

As soon as IIS has read enough of the request, it begins to process the headers. At this point, the unprocessed headers are already in a memory buffer.

Though not documented by Microsoft, I think the first thing done is to extract the HTTP host header, which is then used to obtain configuration information from IIS’ metabase. The host header is vitally important, as all processing and analysis – even the further decoding – is depending on the virtual host being requested. The information from the host header will be used to obtain configuration information during all stages of IIS processing, not just decoding. If there is no (valid) host header, then the default web site is used. If there is no default web site and also no web site configured specifically for those requests without a (valid) host header, the request is simply discarded at this stage. Please note that this also means no log entry will be written as this is done in stage 3.

Then, the final URL is decoded. Depending on the ISAPI filters installed, this can be a length process. In its most basic form, the URL decoder looks into the supplied HTTP host header first and obtains the path mapping from the IIS metabase. Please note that not only the configured home directory plays a role in this calculation but also configured virtual directories, if any. Lastly, ISAPI filters have the ability to change the URL decoding in any way they like. As multiple ISAPI filters are loaded, multiple interim decodes may happen. In any case, at the end of this process the URL is fully decoded and pointing to an actual file system path of a file to execute (or deliver, in case of a static page).

Then, authentication is performed. The result of this operation is the user context into which the request should be served. That context is later on used for all permission checks. Please note that in IIS, other than in Apache, every request is executed in the security ycontext of a specific windows user account. This also applies to anonymous requests. In MMC, you can configure the account used for anonymous requests. By default, this is called IUSR_<Machinename>. There may be other default users, depending on the exact techonology and version used (for example, ASP.NET defines another anonymous user account).

Once the user is know, IIS checks the NTFS ACLs (if any) of the previously decoded actual file to be executed. If the authenticated user does not have proper privileges, IIS denies access to it. Please note that it looks like IIS does this check by itself. It is not relying on the operating system to check if the authenticated user has proper permissions. A possible way to do so would have been to impersonate as the authenticated user and try to access the file. However, IIS at this stage is not yet impersonated as the authenticated user. As such, this mode to check the file permissions seems not to be doable. On the bottom line this means that if there is a bug in IIS’ permission checking, the operating system itself is out of luck and can not detect that. Former vulnerabilities (especially the Unicode Path Mapping yulnerability) prove this observation and also show the damage potential this way of processing has.

As written above, authentication can be quite complex, especially if Microsoft proprietary methods are used. If the user can not properly be authenticated, a “Request denied” page is served back to the requestor. In this case, the request is simply not served, which means stage 2 below is not executed. However, stage three will be used in this case and as such logging of invalidly authenticated requests will happen.

Please note the fine difference: if the authentication fails, IIS continues to work and just does not execute the request. If, however, an vulnerability is exploited during this stage, IIS will probably not continue normally and the request will most probably never be logged.

Once this is done, IIS immediately seems to begin reading further incoming data (for example in a post stream). It looks like this is asynchronous to actual request execution (but I have not verified this).

Stage 2: Serving the Request

Serving the request is easy once the incoming parameters are set.

IIS is a multithreaded application. It has a pool of so-called worker threads. One of these threads is assigned the incoming web request for processing. If all worker threads are currently serving incoming (unfinished) requests, IIS typically just creates a new worker thread to serve the new request. In other cases, IIS does not create a new worker thread but suspends processing of the incoming request until a new worker thread is available. Which method IIS chooses is depending by the IIS configuration, machine utilization and machine resources (and maybe some other factors I don’t know about). I am also not sure if the worker thread is assigned beginning in stage 2 or right at the start of the processing sequence, at the start of stage 1 above. In any case, the actual processing will be carried out in the worker thread, and this is important for our description.

Before actually serving the request, IIS checks what kind of processing it must do. There are three potential cases:

  • Static files
  • Script files
  • Executable files

Static files are served by IIS itself. No other software is involved in serving static files. However, ISAPI filters are still involved while static files are being served, so there is a potential for failure here as well as in core IIS. I deem the potential in core IIS to be very minimalistic. Please note that no external process is started when serving static files.

Script and executable files are processed in more or less the same way, thus I describe them together. An executable file is a file that is executable by itself, like an (CGI) exe file. A script file needs the help of a script processor. Examples of the later are ASP or PHP scripts – and also perl.

A new process needs to be created for all files where the script processor is not available as an in-memory ISAPI DLL (ASP and ASP.NET ar such DLLs). Typical examples for non.resident script processors are PHP and perl. In their case, an external process needs to be created. From IIS’ point of view, it does not really make a difference if the new process is created for a script processor or for a “real” executable. A script processor is just an executable that needs to be passed in some parameters in a specific way.

To create a new process, IIS first creates a new process space. This is still done in the security context of IIS itself, most often the SYSTEM account. Then, IIS loads the image file to be executed into that still-empty process stage. Again, this is done by IIS itself. Consequently, file system audit events 560 of source “Security” in the Windows Security Event Log do show the account IIS is running under as the process initiator. This can be misleading when doing analysis of the event logs. When the image is loaded then IIS impersonates into the context of the authenticated user and changes the new processes access token to be that of the authenticated user. If you are analysing this in the Windows Security Event log, there should be a event id 600, source “Security” that holds this information. IIS then runs the newly created process.

The new process communicates via defined interfaces (e.g. CGI) with IIS and the web client. Depending on the actual interface used, IIS asynchronously reads and writes web client data.

There is at least one difference between scripts and “plain” executables: for scripts, there is a timeout in the IIS configuration. If scripts don’t finish within a configured amount of time, they are aborted (at least ASP scripts, not sure for all potential scripting processors). “Real” executables, do not have any runtime restriction. In lab, we have run ISAPI DLLs (a form of executable) for more than one day in a single IIS thread. IIS did neither abort that executable nor log any error or warning message.

This behaviour can be used in an attack: An attacker exploits a vulnerability that allows him to inject code into one of the worker threads. This code will do malicious things and run in a tight loop (much as SQL Slammer [5]did). As such, it will never finish, thus the worker thread will also never finish. What does this mean? Well, we are stuck at stage two and will never reach stage 3, so no logging will happen.

When the executable (or script processor) is done with the request, it terminates and control returns to the calling worker thread. Please note the process termination will be logged in the correct user context in Windows Security Event log. The event id logged is 593 from source “Security”.

Stage 3: Finishing the Request

Once the request has been served, IIS does some cleanup work. Part of that cleanup work is logging. Now IIS has available all information that it potentially needs to write to the log file. Things like the return status of the process executed, the number of bytes sent or the total execution time. It is for this reason that IIS does not log data at any further stage.

Please note that ISAPI filters can modify IIS’ logging behaviour. If requested by a filter, IIS will call the filter right before the log entry is written. The filter can then decide to modify or even drop the log data.

When the filter returns, IIS finally writes the line to the log entry. This also implies that sequence of events in the log file is based on the time they are finished, not initiated. An example: if request A starts at 9:15:01 and ends at 9:15:05 and request B starts at 9:15:02 and ends at 9:15:03, the sequence in the log file will be B first, then A. The reason for this is that A finished after B and the log file sequence is based on the time the request finished. This fact is worth noting, as the actual processing sequence can not accurately be guessed from web server logs when doing forensic analysis. This most probably applies to all web server logs, not just IIS logs.

Copyright

This document is copyrighted © 2003 by Adiscon GmbH and Rainer Gerhards. Anybody is free to distribute it without paying a fee as long as it is distributed unaltered and there is only a reasonable fee charged for it (e.g. a copying fee for a printout handed out). Please note that “unaltered” means as either the PDF file you (hopefully) acquired or a printout of the same on paper. Any other use requires previous written authorization by Adiscon GmbH and Rainer Gerhards.

If you place the document on a web site or otherwise distribute it to a broader audience, I would appreciate if you let me know. This serves two needs: Number one is I am able to notify you when there is an update available (that is no promise!) and number two is I am a creature of curiosity and simply interested in where the paper pops up.

Credits

Many thanks to Tina Bird of www.loganalysis.org, which had the idea to this paper and provided numerous feedback and tough questions. Without that, it wouldn’t exist at all.

Authors Address

Rainer Gerhards
rgerhards@adiscon.com

Adiscon GmbH
Mozartstrasse 21
97950 Grossrinderfeld
Germany

Disclaimer

The information within this paper may change without notice. Use of this information constitutes acceptance for use in an AS IS condition. There are NO warranties with regard to this information. In no event shall the authors be liable for any damages whatsoever arising out of or in connection with the use or spread of this information. Any use of this information is at the user’s own risk.

Other Formats

This paper is also available in PDF format for easy printing and offline reading.