How to setup File logging in MonitorWare Agent to consolidate Webserver logs and view them in phpLogCon

How to setup File logging in MonitorWare Agent to consolidate Webserver logs and view them in phpLogCon.

Article created 2008-10-05 by Andre Lorbach.

This article will help you to setup an environment where you can store weblogs from your webserver at a central place using MonitorWare Agent, and view and search them using phpLogCon.

You can download a preconfigured configuration from this link and import that into your own system. The configuration sample contains comments for better understanding. MonitorWare Agent Client can import the XML/REG configuration file via the “Computer Menu”.

1. Requirements
1.1 About the requirements
1.2 Installing and configuring WAMP
1.3 Installing MYSQL ODBC Connector
2. Installing and configuring MonitorWare Agent
2.1 Download and Install MonitorWare Agent
2.2 Setup Processing RuleSet
2.2.1 Setup Database Logging
2.2.2 Create the Database Action in MonitorWare Agent
2.3.1 Setup File Logging
2.4 Add File Monitor Service(s)
2.5 Starting MonitorWare Agent and verifying the configuration
3. Install and Setup phpLogCon
3.1 Download and copy phpLogCon to the right location
3.2 Install and configure phpLogCon
Final Thoughts

1. Requirements

1.1 About the requirements

If you already have a web server with PHP support and MYSQL Server running, you can skip step 1.1 and 1.2.

This can be done with Internet Information Server, but this article focuses on using Apache to do the job. There is another article underway which describes IIS.

So in order to setup phpLogCon later, you will need a web server with PHP support and a MYSQL Server with an administration interface. For these tasks, we recommend the following open source applications:

Apache Webserver
PHP
MYSQL (optionally)
phpMyAdmin (optionally)

You can install and configure all these applications separately, but it is much easier to get WAMP for Windows. WAMP means Apache, MYSQL, PHP on Windows and combines all applications with a default configuration. This results in a system which can be used out of the box. So you do not need to worry about the Apache or MYSQL configuration, you just install WAMP first.

Download the latest WAMP Version from here:
http://www.en.wampserver.com/

1.2 Installing and configuring WAMP

After you downloaded WAMP, start the installation and follow the instructions.
Make sure you do not have a web server or MYSQL Server already installed because this could result in conflicts. Most often Microsoft ISS is already installed on the Windows platform. If so, there is no need to install WAMP, but you still need MySQL and php for IIS. This is described in another guide.

I will use the default installation location in this article which is C:\wamp.

Once the Installation is finished, a new Icon appears in Windows Icon tray. Click it, and choose “Localhost” from the menu to verify if the installation was performed successfully. If it was, you should see a web site looking like the one on the right.

To check if your MYSQL is running, click on the phpMyAdmin Menu button in the WAMP Menu, and login with the username “root” and no password – if you are asked for a login.

1.3 Installing MYSQL ODBC Connector

If you intend to store messages in MYSQL database, you need to install the MYSQL ODBC Connector. Otherwise you can skip this step.

MonitorWare Agent will need a MYSQL ODBC driver in a later step in order to write into the MYSQL database. These drivers have to be downloaded and installed separately from here:
http://dev.mysql.com/downloads/connector

If your Windows System is a x64 version, it is important to install the x64 Version of the MySQL Connector driver. As the WinSyslog Service runs as a 64bit application itself, it will need the connector to be 64bit as well.

2. Installing and configuring MonitorWare Agent

2.1 Download and Install MonitorWare Agent

So if you have not done so already, go to www.mwagent.com and download the latest MonitorWare Agent Version. It is always recommended to use the latest Version of MonitorWare Agent. Once the download has completed, go ahead and install it. Depending on your system, a system restart may be needed (but it usually is not)

2.2 Setup Processing RuleSet

Start MonitorWare Agent Client, and skip the First Startup Wizard.

Add a new RuleSet and call it “Store Logdata”.

2.2.1 Setup Database Logging

If you want to store messages inside MYSQL, follow this step.

Click on your WAMP Icon, and open the phpMyAdmin. Now Create a new database called “mwagent”.

Once done, select the newly created database, switch to the “SQL” tab and copy the SQL statements from the textbox below.

CREATE TABLE SystemEvents ( ID int unsigned not null auto_increment primary key, CustomerID bigint, ReceivedAt datetime NULL, DeviceReportedTime datetime NULL, Facility smallint NULL, Priority smallint NULL, FromHost varchar(60) NULL, Message text, NTSeverity int NULL, Importance int NULL, EventSource varchar(60), EventUser varchar(60) NULL, EventCategory int NULL, EventID int NULL, EventBinaryData text NULL, MaxAvailable int NULL, CurrUsage int NULL, MinUsage int NULL, MaxUsage int NULL, InfoUnitID int NULL , SysLogTag varchar(60), EventLogType varchar(60), GenericFileName VarChar(60), SystemID int NULL ); CREATE TABLE SystemEventsProperties ( ID int unsigned not null auto_increment primary key, SystemEventID int NULL , ParamName varchar(255) NULL , ParamValue text NULL )

Now insert the copied commands into the SQL field. Then Click “GO”, you should see “Your SQL query has been executed successfully” after that as well as two new tables on the left list called systemevents and systemeventsproperties.

2.2.2 Create the Database Action in MonitorWare Agent

Get back to the MonitorWare Agent Client and create a new Rule in your self-created RuleSet called “Database”. Then add a new “Write to Database” Action, and name it “MYSQL ODBC”. After creating this action, you should automatically be taken to the actions properties.

Click on the “Data Sources (ODBC)” button to open the System ODBC Administrator. Click on the “System DSN” Tab and add a new Datasource, select “MySQL ODBC 5.1 Driver” as driver. It is important to add a System DSN rather then a User DSN, because User DSN’s are not usable by the MonitorWare Agent Service (this is a Windows design restriction).

Name the new datasource “mwagent” and use “localhost” as Server, “root” as username and no password. Then you are able to select the database which we created before called “mwagent”.

Check the database logging action again, it should look like the one in the screenshot.

2.3.1 Setup File Logging

If you want to consolidate your webserver logfiles in one large logfile, proceed with this step. Otherwise you can skip this step.

Create a new rule called “File Logging” and add a “Write to File” Action.
Select Custom Line Format, use the following (use copy&paste to enter it):%timegenerated:::localtime%,%timereported:::localtime%,%source%,%syslogfacility%,%syslogpriority%,%syslogtag%: %msg%%$CRLF%

2.4 Add File Monitor Service(s)

First of all find the exact location of the webserver logfiles you want to monitor. In this example, I use the access.log from my local wamp installation located at C:\wamp\logs.

Now add a file monitor service. Inside that service, configure the correct file and path name of the file to be monitored. If you would like to read multiple files, you can enable the property replacer inside the filename.

Be sure to specify the log file type to “W3C WebServer Logfile”. This is important so that the contents can properly be interpreted.

It may also be a good idea to set a syslog tag name that matches the filename (or the function of the file name, e.g. “online-shop”). By doing so, you can easily filter inside phpLogCon.

2.5 Starting MonitorWare Agent and verifying the configuration

From the MonitorWare Agent configuration point of view, everything is setup now. So kindly start the MonitorWare Agent Service and wait a few moments, so that the data can be processed.

If you are using file logging, you should see that the logs folder on C:\ has been created and contains a WebServer.log file. If not, something went wrong. In this case please check the Windows Application EventLog for possible error reports from MonitorWare Agent.

If you are logging into a database, switch back to phpMyAdmin and browse through the systemevents table. You should see at least one data record in this table now, like in the screenshot sample. If not, something went wrong, in this case please check the Windows Application Event Log for possible error reports from MonitorWare Agent.

3. Install and Setup phpLogCon

3.1 Download and copy phpLogCon to the right location

If you are using MonitorWare Agent 8.3 or higher, a proper version of phpLogCon can be found in the MonitorWare Agent installation folder. If you are using an older Version of MonitorWare Agent, I recommend to download the latest stable or beta build from here: http://www.phplogcon.org/downloads
In this article I will use phpLogCon Version 2.5.13.

To unpack the install set, you need a program capable of processing tar.gz files. Most ZIP programs support this. If you do not have one, you can find WinRAR by following the link (we have no affiliation with the makers of WinRAR, but have found it to be a useful tool – use at your own risk).

Open windows explorer and go to the www folder of your Apache web server, which is the folder where you can place html/php files. By default this will be “C:\wamp\www” if you have installed WAMP into the default installation folder. Create a new folder called phplogcon there.

If you downloaded and unpacked phpLogCon, and copy or move the content of the src folder into the C:\wamp\www\phplogcon folder. If you have MonitorWare 8.3 or higher, you can use and copy the contents of the phpLogCon folder of your MonitorWare installation.

The explorer window should look like in the screenshot now.

3.2 Install and configure phpLogCon

Open this link to start the phpLogCon installation: http://localhost/phplogcon/

If you do not see a page like in the screenshot, something went wrong in the steps before, please check them in this case.

Otherwise click on the text-link “here” on phpLogCon’s error page to start its installation routine.

Follow the installation steps of phpLogCon.

I recommend to “Enable User Database” in Step 3, as this will give you an advanced admin control panel. The User Database requires a MYSQL database to work, you can use the same one as you are using for MonitorWare Agent.

If you are using MYSQL to store log messages and you have reached Step 7, switch the source type to “MYSQL Native” and name the Source “WebLogStore DB” Use “mwagent” as Database Name and “root” as Database User. Leave the other configuration variables as they are, see the screenshot for how it should look like. Then click on the Next button to finish the installation.

After you finished the Installation of phpLogCon, you need to login and switch to the sources admin and configure the source “WebLogStore DB” there.

– In field “Message Parsers” add apache2 if you are using combined log format. Add apach2common if you are using common log format.

If you are using file logging and you have reached Strep 7, switch the source type to “Diskfile” and name the Source “WebLogStore FILE” Use “C:/logs/WebServer.log” as syslog file. Leave the other configuration variables as they are, see the screenshot for how it should look like. Then click on the Next button to finish the installation.

After you finished the Installation of phpLogCon, you need to login and switch to the sources admin and configure the source “WebLogStore FILE” there.

– In field “Message Parsers” add apache2 if you are using combined log format. Add apach2common if you are using common log format.

After clicking on the “Finish” link, you should see a working phpLogCon installation. If you do not see any data, there may be no data in your database yet. Otherwise you will see an error code and message from phpLogCon.

Final Thoughts

I hope this article will help you installing and configuring phpLogCon and MonitorWare Agent. If you have problems or question related to this article, don’t hesitate to contact me or our support by email.

Please note that while this setup works, it is not very secure. At a minimum, it is recommended to set proper passwords for the databases (instead of using a password-less root account). Please review the relevant documentation on how to do that.

Forwarding NetApp Event Log Entries via Syslog

Article created 2008-09-22 by Rainer Gerhards

NetApp devices provide diagnostic information via an Windows Event Log like interface. This enables their messages to be browsed by Windows Event viewer and and be automatically processed by tools like EventReporter and MonitorWare Agent.

It goes without saying that there are ample benefits from this capability. For example, instant alerting can be enabled by simply reading the NetApp event logs, checking for some conditions and alerting (for example via email) if something unusual is logged. Another big benefit is that by converting the NetApp Event Logs into syslog, they can be forwarded to a central loghost and thus easily be integrated into a centralized logging and alerting system.

There are two ways to access the NetApp logs. First of all, NetApp exposes them as files which obey the event log file format. These files are dynamically re-written and as such considered to be “live”. Adiscon products support this mode since long and are used with great success in that mode (there exists another article on how to do that).

Secondly, NetApp also provides an emulation of the Windows event log API. Under that emulation, a NetApp devices “looks and feels” just like another Windows machine offering its event log. Well … mostly. There are a couple of subtle differences, with the most important one being that the event log record number is not incremented on NetApp devices. This record number is associated with each event log record. Native Windows always increments this number (up to a very high max value, where it then wraps). The NetApp emulation does not use such a unique record number. Instead, the API always exposes a fixed number of most recent records, but always uses the same recrord number for them. So while the data is being rolled as new records come in, the record number stays the same. Thus, for example, if new data comes in, the data for record number 312 is moved to record number 311 and 312 receives the data from record 313. Under this implementation, event log record numbers do no longer identify a given event log entry.

Adiscon products are optimized for highest throughput and least demand on system ressources. One way to do that is to utilize the uniqueness of record numbers. This optimization leads to problems on NetApp. Namely, new records are never forwarded, because no new record numbers are seen (which leads EventReporter and MonitorWare Agent to beliefe there are no new records). The cure, however, is simple. First of all, checksum verification for the last processed event must be turned on in the configuration of the event log monitor in question. This ensure that an internal checksum is generated as a unique identifier for an event log record. Secondly, the event log monitor must be instructed to always search for new events based on that checksum. This will enable the monitor to properly detect new events. This configuration can be seen in the screenshot below:

Event Log Monitor settings for NetApp

Please note that while we turn off some optimizations with these settings, performance is still quite good. Most importantly, typical NetApp event logs are relatively small as the contain only a small subset of recent records. This makes reading through them quite painless. Secondly, we still apply a lot of other optimizations, especially when comparing to other tools which blindly try to re-do everything always. Users are still advised to use a sensible sleep value for the event log monitor (the default should be fine), as without the optimizations additional i/o overhead is unavoidable.

Should you have problems implementing this configuration, please contact support@adiscon.com for assistance. We are glad to help.

How to monitor and forward message tracking logfiles from Microsoft Exchange Server using MonitorWare Agent

How to monitor and forward message tracking logfiles from Microsoft Exchange Server using MonitorWare Agent.

Article created 2008-05-09 by Andre Lorbach.

This article will guide you in how to monitor and forward message tracking logfiles from Microsoft Exchange Server using syslog over tcp. As receiver you have the choice of using different applications like WinSyslog, MonitorWare Agent or even open source projects like rsyslog.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent client can import the XML/REG configuration file by using the “Computer Menu”.

If message tracking is enabled in Exchange Server, the logfiles are created daily by default and contain informations about each message which is routed through the Exchange server. Depending on the workload of your server, you may need to delete the logfiles from time to time to save harddisk space. But you need a backup of these logfiles in case you need to review them. This is where MonitorWare Agent comes into play. The File Monitor of MonitorWare Agent can be easily used to forward the message tracking logfiles to a syslog repository.

1. Exchange Server preparations
1.1 Enabling Message tracking logging in Exchange Server
2. Installing and Configuring MonitorWare Agent
2.1 Download and Install MonitorWare Agent
2.2 Setup up basic MonitorWare Agent configuration
2.3 Configure the Foward Syslog Action.
Final Thoughts

1. Exchange Server preparations

1.1 Enabling Message tracking logging in Exchange Server

If already enabled, you can skip this step.

To enable message tracking logging, kindly check the “Enable message tracking” option.

Optionally you can select “Remove log files” option and define how old the logfiles have to be in order to get deleted. You may also change the log file directory, may be to another hard disk, in order to save the space on the main hard disk.

If enabled, Exchange Server creates one message tracking logfile per day. As you can see in the date modified column, Exchange Server is using UTC and not localtitme to create the logfiles. My testmachine is running with European central time which is GMT+1 and GMT+2 daylight saving time (which is currently set).

2. Installing and Configuring MonitorWare Agent

2.1 Download and Install MonitorWare Agent

So if you haven’t done so already, go to www.mwagent.com and download the latest MonitorWare Agent version. It is always recommended to use the latest Version of MonitorWare Agent. Once the download is done, go ahead and install it. You may have to restart after installation, this depends on your system.

2.2 Setup up basic MonitorWare Agent configuration

Start the MonitorWare Agent client and skip the wizard on startup. First we create new “File Monitor” service and name it “Exchange File Monitor”.

Then use the browse button to select the directory which contains the message tracking logfiles. Kindly select one of the logfiles and replace its name with this string: %Y%m%d.log

This will automatically match the logfiles each day, %Y will match the year, %m the month and %d the current day.

It is also important to select UTC as Timemode for the filename, as I already mentioned in step 1.1, the Exchange server used UTC (GMT) to create the logfiles.

Now click on the Advanced Options, and the following dialog will appear. In this dialog, enable the option “ignore empty lines”. The message tracking logfiles sometimes contain empty lines between the logfiles, so this option will remove them automatically.

Make sure that you use only “\n” as message separation sequence, as the typical Windows “\r\n” is not used in the message tracking logfiles.

2.3 Configure the Foward Syslog Action.

The last step is to configure the forward syslog action, first create a new Rule and name it ForwardSyslog. Then create a new Forward Syslog Action and call it Repository for example. You also see a InterActive Action in the sample screenshot here, this is a helper action which forwards to the local InterActive Syslogviewer, which is also installed with MonitorWare Agent by default.

As I wrote in the beginning of this article, there are several syslog products available which can be used as receiver. On Windows, we recommend to use WinSyslog or MonitorWare Agent. On Unix based systems, we recommend to use rsyslog, which is open source of course ;)! Rsyslog is available on many plattforms and integrated in many package systems.

All these syslog products are able to receive syslog message over tcp and also in a persistent connection.

Final Thoughts

I hope this article will help you solving your tasks and shows you the potential of MonitorWare Agent, and what you can archive with it. Feel free to email me for recommendations or questions.

How to send EventLog entries as SNMP Traps with MonitorWare Agent

How to send EventLog entries as SNMP Traps with MonitorWare Agent.

Article created 2008-03-06 by Andre Lorbach.

This article will guide you to use MonitorWare Agent to generate SNMP Traps from EventLog entries and send them to your SNMP management software. This article also requires at least MonitorWare Agent 5.2 or higher, and the custom ADISCON mibs which are included since MonitorWare Agent 5.2.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent Client can import the XML/REG configuration file by using the “Computer Menu”.
To obtain the most recent custom ADISCON mib files, download these two files und put them into your mibs directory of your MonitorWare Agent installation.
ADISCON-MIB.txt
ADISCON-MONITORWARE-MIB.txt

1. Configuring MonitorWare Agent
1.1 Download and Install MonitorWare Agent
1.2 Setup a EventLog Monitor in MonitorWare Agent
2. Configuring the SNMP Trap
2.1 Create SNMP Trap Action
2.2 Filtering for EventLog severity (Optional)
2.3 Start sending SNMP Trap

1. Configuring MonitorWare Agent

1.1 Download and Install MonitorWare Agent

So if you haven’t done so already, go to www.mwagent.com and download the latest MonitorWare Agent Version. It is always recommended to use the latest Version of MonitorWare Agent. Once the Download is done, go ahead and install it. You may have to restart after installation, this depends on your System.

1.2 Setup a EventLog Monitor in MonitorWare Agent

Start the MonitorWare Agent Client and skip the wizard on startup.

Then add a new EventLog Monitor called “Main EventLog Monitor”. I have set the Sleep time to 5 seconds, for testing purposes. But you can also set this value to 5 seconds in production, it won’t have much impact on the Servers performance.

You also might unselect EventLog Types you do not want to monitor, for this article I will allow all EventLog Types.

2. Configuring the SNMP Trap

2.1 Create SNMP Trap Action

Now add a new Rule under your Default RuleSet called SendTrap. Then add a Send SNMPTrap Action. The default values will already generate a generic “monitorwaretrap”, which is fine for most cases. But we are going to configure our own trap properties.

So you have noticed that the Trap OID and the variable OID’s are represented numeric. Once you click on the Browser Button, the Client will automatically load and display the installed mibs. You can configure the Configuration Client to automatically load the mibs during each startup in the Client Options.

So as you can see you have a few trap OID’s available, in this article we will use the eventmontrap OID which is “.1.3.6.1.4.1.19406.1.2.3”, or in human readable form “ADISCON-MONITORWARE-MIB::eventmontrap”. You can actually define the one or the other form as OID, both will work but the textual representation only if you have the ADISCON Mibs installed. The the numeric representation is always the saver way to configure the OID’s.

Now what you don’t see in the mib browser is the list of variables which are connected with the SNMP Trap. For the eventmontrap, we need a few snmp variables:

genMsg,
genSource,
eventlogEventID,
eventlogEventType,
eventlogEventSource,
eventlogEventSeverity,
eventlogEventCategoryID,
eventlogEventCategoryName,
eventlogEventUser

Start removing the default configured variable, and add our own ones (as in the list above). Add one variable, and use the Mib Browser to select the suitable OID’s and also the correct variable values (See the screenshot for more).

2.2 Filtering for EventLog severity (Optional)

With our current setup, you would send one SNMP Trap for each incoming Syslog messages. But you may not want this, so you can optionally add some filters to reduce the number of outgoing SNMP Traps.

For example you can add a Syslog Severity (Priority) filter, so that only EventLog entries with error messages will be send as trap to your SNMP Manager.

2.3 Start sending SNMP Trap

Now you are ready to start the MonitorWare Agent, note that you properly will get a lot of SNMP Events during the first run.

To show you how the result looks like, here is the output of snmptrapd on a linux machine. There are many SNMP Manager utilities out there, you can even receive SNMP Traps with MonitorWare Agent itself if you like.

2008-03-07 15:18:31 172.16.0.122 [UDP: [172.16.0.122]:1119]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (742090390) 85 days, 21:21:43.90 SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.19406.1.2.1 SNMPv2-SMI::enterprises.19406.1.1.2.1 = STRING: “MWAgent: This is a Test no. 1” SNMPv2-SMI::enterprises.19406.1.1.2.2 = INTEGER: 3 SNMPv2-SMI::enterprises.19406.1.1.2.3 = INTEGER: 16

When you receive the trap with MonitorWare Agent, the message output will look like this:

MonitorWare: source=”172.16.0.122″ community=”public” version=”Ver2″ variables: snmp_var_1 = ‘DISMAN-EVENT-MIB::sysUpTimeInstance: ‘Timeticks: (741870389) 85 days, 20:45:03.89” , snmp_var_2 = ‘SNMPv2-MIB::snmpTrapOID.0: ‘OID: ADISCON-MONITORWARE-MIB::syslogtrap” , snmp_var_3 = ‘ADISCON-MONITORWARE-MIB::syslogMsg: ‘STRING: “MWAgent: This is a Test Error MEssage no. 1″” , snmp_var_4 = ‘ADISCON-MONITORWARE-MIB::syslogSeverity: ‘INTEGER: error(3)” , snmp_var_5 = ‘ADISCON-MONITORWARE-MIB::syslogFacility: ‘INTEGER: local0(16)”

As you can see eventlogEventCategoryID and eventlogEventCategoryName are missing. Most EventLog entries do not have a Event Category assigned, so these variables are not added into the SNMP Trap.

Final Thoughts

I hope this article will help you solving your tasks or shows you the potential of MonitorWare Agent, and what you can archive with it. Feel free to email me for recommendations or questions. Of course, the outlined actions are only samples and you may do other things with them.

How to send generic SNMP Traps with MonitorWare Agent

How to send generic SNMP Traps with MonitorWare Agent.

Article created 2008-03-06 by Andre Lorbach.

This article will guide you to use MonitorWare Agent to generate generic SNMP Traps and send them to your SNMP management software. It is also possible to use WinSyslog instead of MonitorWare Agent in some cases, however this article will target the more powerful MonitorWare Agent. This article also requires at least MonitorWare Agent 5.2 or higher, and the custom ADISCON mibs which are included since MonitorWare Agent 5.2.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent Client can import the XML/REG configuration file by using the “Computer Menu”.
To obtain the most recent custom ADISCON mib files, download these two files und put them into your mibs directory of your MonitorWare Agent installation.
ADISCON-MIB.txt
ADISCON-MONITORWARE-MIB.txt

As you know MonitorWare Agent has many different input sources (services) from which we can generate useful traps. In this article, I will show you how to generate a SNMP Trap from a received Syslog message, so we are going to use the Syslog Service.

1. Configuring MonitorWare Agent
1.1 Download and Install MonitorWare Agent
1.2 Setup a Syslog Server in MonitorWare Agent
2. Configuring the SNMP Trap
2.1 Create SNMP Trap Action
2.2 Filtering Syslog messages (Optional)
2.3 Sending a test SNMP Trap

1. Configuring MonitorWare Agent

1.1 Download and Install MonitorWare Agent

1.2 Setup a Syslog Server in MonitorWare Agent

Start the MonitorWare Agent Client and skip the wizard on startup.

Then add a new Syslog Service called “Main Syslog Server”. We use default values here, Port 514 UDP. Leave the other Options as they are.

2. Configuring the SNMP Trap

2.1 Create SNMP Trap Action

First add a new Rule under your Default RuleSet called SendTrap. Then add a Send SNMPTrap Action. The default values will already generate a generic “monitorwaretrap”, which is fine for most cases. But we are going to configure our own trap properties.

So as you can see you have a few trap OID’s available, in this article we will use the syslogtrap OID which is “.1.3.6.1.4.1.19406.1.2.1”, or in human readable form “ADISCON-MONITORWARE-MIB::syslogtrap”. You can actually define the one or the other form as OID, both will work but the textual representation only if you have the ADISCON Mibs installed.

Now what you don’t see in the mib browser is the list of variables which are connected with the SNMP Trap. For the syslogtrap, we need syslogMsg, syslogSeverity and syslogFacility.

So we are going to remove the default configured variable, and add our own ones, for the message, syslog severity and syslog facility. Kindly add 3 new variables, you use the Mib Browser to select the suitable OID’s and also the correct variable values (See the screenshot for more).

2.2 Filtering Syslog messages (Optional)

With our current setup, you would send one SNMP Trap for each incoming Syslog messages. But you may not want this, so you can optionally add some filters to reduce the number of outgoing SNMP Traps.

For example you can add a Syslog Severity (Priority) filter, so that only syslog error messages will be send as trap to your SNMP Manager.

2.3 Sending a test SNMP Trap

The easiest way to create a SNMP Trap for testing now is use the “Send Syslog Test Message” from the Configuration Client tools menu. If you configured filters, don’t forget to set the correct syslog facility and priority.

2008-03-07 15:18:31 172.16.0.122 [UDP: [172.16.0.122]:1119]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (742090390) 85 days, 21:21:43.90 SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.19406.1.2.1 SNMPv2-SMI::enterprises.19406.1.1.2.1 = STRING: “MWAgent: This is a Test no. 1″ SNMPv2-SMI::enterprises.19406.1.1.2.2 = INTEGER: 3 SNMPv2-SMI::enterprises.19406.1.1.2.3 = INTEGER: 16

When you receive the trap with MonitorWare Agent, the message output will look like this:

Final Thoughts

Monitor a SNMP Device using MonitorWare Agent

Article created 2008-02-26 by Andre Lorbach.

This article will help you to monitor SNMP capable devices using the new SNMP Monitor Service of MonitorWare Agent. There are many devices out there which support SNMP, and can also be queried for information’s using SNMP GET. We will use SNMP GET to monitor a device, if we get a respond, the device is most likely running. If we do not get a response, the device may be offline or is powered off. We will use the Send Email Action to generate Alert Emails in case of that a monitored device is not responding anymore.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent Client can import the XML/REG configuration file by using the “Computer Menu”.

As I said earlier there are many devices out there which do support SNMP like printers, routers, managed switches, linux / windows server and so on. In this article I will show you how to setup the SNMP Monitor Service to monitor a HP LaserJet 4000 laser printer.

1. SNMP Device (Printer) Setup
1.1 Configuring the Printer
2. Configuring MonitorWare Agent
2.1 Download and Install MonitorWare Agent
2.2 Setup Basics in MonitorWare Agent
2.3 Create a Forwarding Rule for the InterActive SyslogViewer (Optional!)
2.4 Create an Email alert
2.5 Configure Filters for the Email Alert
Final Thoughts

1. SNMP Device (Printer) Setup

1.1 Configuring the Printer

As far as I know this printer does not need any special setting to have SNMP support enabled. So by default SNMP is enabled, and it is possible to overwrite the used snmp community name. However for this article I will not modify this setting.

The only thing we will need for this device is it’s IP Address which is 172.16.0.15.

2. Configuring MonitorWare Agent

2.1 Download and Install MonitorWare Agent

2.2 Setup Basics in MonitorWare Agent

Start the MonitorWare Agent Client and skip the wizard on startup. First we create new “SNMP Monitor” Service by right clicking the Configured Services node and going to the Add Service menu.

Insert the IP of the device you want to monitor in the remote host field. You can leave the default values for the other configuration options, they will work fine for most devices.

The Query OID I use in this sample will query the system name of the device. However there are several other variables you pick for monitoring such as:

.1.3.6.1.2.1.1.6 (i.o.d.i.mgmt.mib-2.system.sysLocation)
.1.3.6.1.2.1.1.5 (i.o.d.i.mgmt.mib-2.system.sysName)
.1.3.6.1.2.1.1.4 (i.o.d.i.mgmt.mib-2.system.sysContact)
.1.3.6.1.2.1.1.3 (i.o.d.i.mgmt.mib-2.system.sysUpTime)
.1.3.6.1.2.1.1.2 (i.o.d.i.mgmt.mib-2.system.sysObjectID)
.1.3.6.1.2.1.1.1 (i.o.d.i.mgmt.mib-2.system.sysDescr)

Other OID’s might also be available, it depends on device you are monitoring. There is also a Instance subidentifier option available. I recommend to leave this value to 0, it is only useful if you want to query a OID which contains multiple data.

2.3 Create a Forwarding Rule for the InterActive SyslogViewer (Optional!)

This is an optional step, only useful for testing and debugging the SNMP Monitor. You can disable the Action of this rule later if you want. As we are using the UDP protocol to forward syslog messages locally, it doesn’t really matter.

So first of all create a new Rule called “FwSyslog” and add a new Forward Syslog Action. The Syslog Server is 127.0.0.1 and the syslog port is 10514. See the screenshot for more details.

2.4 Create an Email alert

The best option to get alerted is by email. So we create another rule called EmailAlert and add a Forward Email Action. Please fill Sender, Recipient and Mailserver configuration yourself.

Use the following text as mail subject:
SNMP Monitor Check for ‘%source%’ failed

Use the following text as message format:
SNMP Monitor Check failed for ‘%source%’ SNMP Community: %snmp_community% SNMP Checked OID: %snmp_checkoid% SNMP Status: %snmp_status% Complete Message: %msg%

2.5 Configure Filters for the Email Alert

We were not finished yet ;). We need to configure some filters, otherwise you would get one Email for each SNMP Monitor check, even if successfully.

So add a new Custom Property filter, with the property name “%snmp_status%”. Use the compare operation “not equal” and Property Valur of “0”. So this means the Actions in this rule will be fired whenever the status is not 0, and every status which is not 0 means there was an error.

To avoid email flooding, set the Minimum WaitTime to 600 seconds. This means it doesn’t matter how failures are generated, in 10 minutes there will only be one email alert.

Final Thoughts

How to monitor a Software-Raid on Windows 2003 by using the EventLog Monitor of MonitorWare Agent

How to monitor a Software-Raid on Windows 2003 by using the EventLog Monitor of MonitorWare Agent.

Article created 2008-01-17 by Andre Lorbach.

This article will guide you in how to monitor a software raid on Windows 2003 by filtering specific events by using the EventLog Monitor in MonitorWare Agent. This is also possible with EventReporter, however this article will target the more powerful MonitorWare Agent.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent Client can import the XML/REG configuration file by using the “Computer Menu”.

Raid Systems have a big advantage for failover support and prevent data loss. But what when a hard disk is failing, you don’t know it? Windows Server Systems often run for months without being monitored, and what if two hard disk fail in this time period? A nightmare for every system administrator. So we will setup a EventLog Monitor in MonitorWare Agent which alert you by email in case of a raid brakes, a hard disk fails or anything else bad happens.

1. Creating a Windows Software Raid (Skip if Raid exists!)
1.1 Convert Hard disks into dynamic disks
1.2 Adding a Mirror to the existing system partition
2. Installing and Configuring MonitorWare Agent
2.1 Download and Install MonitorWare Agent
2.2 Setup up basic MonitorWare Agent configuration
2.3 How to verify that the alert is working?
Final Thoughts

1. Creating a Windows Software Raid (Skip if Raid exists!)

1.1 Convert Hard disks into dynamic disks

So in case you have no Software Raid configured yet, open the Computer Management und go to the Disk Management. You will see your System drive and you should have a second hard disk with enough free space available. For a sample see the screenshot.

Right-Click one of the disks and click on “Convert to Dynamic Disk”. A wizard will appear, select both hard disks, the system one and the one you are going to use as raid mirror. Once you have accepted this, a couple of questions will follow which you need to accept and finally a reboot is required. This is because Windows can not convert a hard disk if the system is running on it.

Once you have rebooted, logged in and open the Disk Management. You will notice the different partition color. This means your system partition runs on a dynamic disk now, the conversion went fine. If not review the System EventLog for possible errors.

1.2 Adding a Mirror to the existing system partition

All requirements for a software raid (mirror) are now given, so kindly right click your system partition and click on “Add Mirror“. A requester will open which will ask you on which disk you want to add the mirror. In our sample, this would be disk 1. After the mirror has been added, Windows will start regenerating the mirror which means it will sync both hard disks. This may take some time depending on the size of your hard disk, maybe even hours.

As you can see the partitions are now marked red which represents the color for mirrored partitions. After the synchronization has finished, the red partitions will be marked as healthy in the Disk Management view.

2. Installing and Configuring MonitorWare Agent

2.1 Download and Install MonitorWare Agent

2.2 Setup up basic MonitorWare Agent configuration

Start the MonitorWare Agent Client and skip the wizard on startup. First we create new “Event Log Monitor” Service. Uncheck all event log types except System, as this is the only event log needed to achieve our goal. If you like to monitor other Event Log Types too, you may select them. It will have no impact on our following configuration.

Now we can add another Rule called “Send Email Alert”. This rule will have a few filters to only allow events with warning or error severity. The Eventlogtype is System and the event sources which matter to us are dmio and dmboot. The filters should look like in this screenshot.

For additional reference, here is a list of possible dmboot und dmio events:
Event ID 1: “dmboot: Volume %2 (no mountpoint) started in failed redundancy mode.”
Event ID 2: “dmboot: Volume %2 (%3) started in failed redundancy mode.”
Event ID 3: “dmboot: Failed to start volume %2 (%3)”
Event ID 4: “dmboot: Failed to encapsulate selected disks”
Event ID 5: “dmboot: Disk group %2 failed. All volumes in the disk group are not available.”
Event ID 6: “dmboot: Failed to auto-import disk group %2. All volumes in the disk group are not available.”
Event ID 7: “dmboot: Failed to restore all volume mount points. All volume mount points may not be available. %2”

Event ID: 1, “dmio: Device %2,%3: Received spurious close”
Event ID: 2, “dmio: Failed to log the detach of the DRL volume %2”
Event ID: 3, “dmio: DRL volume %2 is detached”
Event ID: 4, “dmio: %2 error on %3 %4 of volume %5 offset %6 length %7”
Event ID: 5, “dmio: %2 %3 detached from volume %4”
Event ID: 6, “dmio: Overlapping mirror %2 %3 detached from volume %4”
Event ID: 7, “dmio: Kernel log full: %2 %3 detached”
Event ID: 8, “dmio: Kernel log update failed: %2 %3 detached”
Event ID: 9, “dmio: detaching RAID-5 %2”
Event ID: 10, “dmio: object %2 detached from RAID-5 %3 at column %4 offset %5”
Event ID: 11, “dmio: RAID-5 %2 entering degraded mode operation”
Event ID: 12, “dmio: Double failure condition detected on RAID-5 %2”
Event ID: 13, “dmio: Failure in RAID-5 logging operation”
Event ID: 14, “dmio: log object %2 detached from RAID-5 %3”
Event ID: 15, “dmio: check_ilocks: stranded ilock on %2 start %3 len %4”
Event ID: 16, “dmio: check_ilocks: overlapping ilocks: %2 for %3, %4 for %5”
Event ID: 17, “dmio: Illegal vminor encountered”
Event ID: 18, “dmio: %2 %3 block %4: Uncorrectable %5 error”
Event ID: 19, “dmio: %2 %3 block %4:\r\n Uncorrectable %5 error on %6 %7 block %8”
Event ID: 20, “dmio: Cannot open disk %2: kernel error %3”
Event ID: 21, “dmio: Disk %2: Unexpected status on close: %3”
Event ID: 22, “dmio: read error on object %2 of mirror %3 in volume %4 (start %5, length %6) corrected”
Event ID: 23, “dmio: Reassigning bad block number %2 on disk %3”
Event ID: 24, “dmio: Reassign bad block(s) on disk %2 succeeded”
Event ID: 25, “dmio: Fail to reassign bad block(s) on disk %2: error 0x%3”
Event ID: 26, “dmio: Found a bad block on disk %2 at block number %3”
Event ID: 27, “dmio: Corrected a read error during RAID5 initialization on %2”
Event ID: 28, “dmio: Failed to recover a read error during RAID5 initialization on %2: error %3”
Event ID: 29, “dmio: %2 read error at block %3: status 0x%4”
Event ID: 30, “dmio: %2 write error at block %3: status 0x%4”
Event ID: 31, “dmio: %2 write error at block %3 due to disk removal”
Event ID: 32, “dmio: %2 read error at block %3 due to disk removal”
Event ID: 33, “dmio: %2 is disabled by PnP”
Event ID: 34, “dmio: %2 is re-online by PnP”
Event ID: 35, “dmio: Disk %2 block %3 (mountpoint %4): Uncorrectable read error”
Event ID: 36, “dmio: %2 %3 block %4 (mountpoint %5): Uncorrectable read error”
Event ID: 37, “dmio: Disk %2 block %3 (mountpoint %4): Uncorrectable write error”
Event ID: 38, “dmio: %2 %3 block %4 (mountpoint %5): Uncorrectable write error”

The next step is to create a SendEmail Action and configure it like in the screenshot.

Here is the Event message we suggest to use, but feel free to create and modify your own:
A Software Raid Event has occured: %severity% Server: %source% Reported Time: %timereported% Event message: %msg%

You need to replace the mail server, sender and recipient with yourself.

2.3 How to verify that the alert is working?

There is a simple way to test if our alerting is working, however it isn’t without risks. I only recommend you to do this step if your really want to test the alerting! I do NOT recommend to perform this test on a productive system!

First of all shutdown the server and open the case. Then disconnect the second hard disk by removing the power or the data connector. Then boot the server, once windows is starting the services you should get an alert by email. It should look like the sample email in the screenshot.

If the test was successful, you can shutdown your server again. Connect the power / data connector and boot your server. You may receive the same email message again, as the raid is now OUT OF SYNC. So you need to open the Disk Management and right click the disk with the exclamation mark. Then select “Reactivate Disk”, the raid will begin resynchronization immediately after this.

Final Thoughts

I hope this article will help you solving your tasks and shows you the potential of MonitorWare Agent, and what you can archive with it. Feel free to email me for recommendations or questions.

How to audit File / Directory delete Operations on a Windows System using security auditing

How to audit File / Directory delete Operations on a Windows System using security auditing.

Article created 2007-12-12 by Andre Lorbach.

This article will guide you in how to setup Windows and MonitorWare Agent to track file and directory deletion processes. It is also possible to use EventReporter instead of MonitorWare Agent, however this article will target the more powerful MonitorWare Agent. The guide works both on Workstation and Server versions of Windows.

You can download a preconfigured configuration from here, which you can import on your target system. The configuration sample will have comments for better understanding. The MonitorWare Agent Client can import the XML/REG configuration file by using the “Computer Menu”.

A typical scenario of a small to middle sized company. The employees access a file server and store their documents on it. Suddenly an important document is missing. Who deleted it? And more important when was the file deleted? Oh two months ago? To bad our backups don’t go that far, the document is gone forever … ! Wouldn’t it be great to know if a file is deleted, when it was deleted and also who did delete the file? Maybe it was the revenge of an employee who got fired a few weeks ago?

Using Windows Security Auditing and the EventLog Monitor of MonitorWare Agent, you can set up an environment where you exactly know if a file is deleted, when it is deleted and by whom it is deleted.

1. Windows Settings
1.1 Turn on Security Auditing
1.2 Add Auditing to the folders / drives you want to monitor
1.3 Configure Windows Security EventLog Size
1.4 Testing File Deletion
2. Configuring MonitorWare Agent
2.1 Download and Install MonitorWare Agent
2.2 Setup Basics in MonitorWare Agent
2.3 Use File Logging to store File Deletion
2.4 Create an Email alert

1. Windows Settings

1.1 Turn on Security Auditing

Our first step is to enable “Audit Object access” in the Audit Policy. If your machine is a Workstation like Windows XP / Vista, open the “Local Security Settings” from the “Control Panel->Administrative Tools”. If you want to enable security auditing for the whole domain, or only domain controllers, open the “Domain Security Policy” or “Domain Controller Security Policy” from the “Administrative Tools”. In this article, we will use the “Local Security Settings” on a Windows XP machine.

Enable the “Audit object access” policy here. This will also enable object logging we do not need, but the filters in MonitorWare Agent will deal with these events later.

1.2 Add Auditing to the folders / drives you want to monitor

Now we can configure the Security properties for the drives or directories we want to monitor for file / directory deletions. In order to do so, right click the object you want to monitor and click on properties. In our sample, we will use the whole drive C:\. Once you switched to the security tab, click on the “advanced” button.

This will open advanced configuration options, switch to the “Auditing” tab and click on the “Add” button. Select “Successful” for the “Delete” and “Delete Subfolder and Files“, and also for “Failed” if you want to monitor failed deletion attempts as well. Once you have done this and confirm your changes by clicking on “Apply“, it may take a while until the system begins to record audit records. Good time to get a cup of coffee.

If you want to monitor other file operations, go ahead and activate them. However you will need to enhance the filter rules for MonitorWare Agent later by yourself.

1.3 Configure Windows Security Event Log Size

The System defaults are different from Windows to Windows Version. We need to make sure that event logging is continuous, otherwise the event log will fill up at some time, and no events can be written into it anymore. In our example, we will use 16MB of Maximum log size (Can be up to 256MB) and “Overwrite events as needed“.

1.4 Testing File Deletion

For this sample, we created a folder called “sharing” on the drive C:\ and shared it as a network folder. Then we added a file called document.doc into this folder, and deleted it. All done over the network share.

Then we open the Computer Management Console, and take a look into the Windows Security EventLog. You will see that a bunch of events is being generated for each file we delete. The Event which is interesting for us is the first Event with ID 560. We will filter this event by certain parameters later using MonitorWare Agent. This event contains all information’s we need to determine which file was deleted, when it was deleted and by whom.

2. Configuring MonitorWare Agent

2.1 Download and Install MonitorWare Agent

2.2 Setup Basics in MonitorWare Agent

Start the MonitorWare Agent Client and skip the wizard on startup. First we create new “Event Log Monitor” Service. Uncheck all event log types and only select “Security“, as this is the only event log needed to achieve our goal. Then save the settings.

Now we need to create some basic rules and filter to preprocess the event log entries. The first rule we create is called “DiscardProcessing” and will discard events if certain filters match. We only want to process Events with ID 560. Then there is the Param14 property which will contain “Accesses” masks from the event entry. The event log message will show DELETE, Read_Control and so on, but the parameters actually are RAW which means we have to deal with numbers.

Here is the list for the most common access mask numbers and their meaning:
1537 = Delete
1538 = Read_CONTROL
1541 = synchronize
4416 = ReadData(or List Directory)
4417 = WriteData(or Add File)
4418 = AppendData (or AddSubdirectory or CreatePipeInstance)
4419 = ReadEA
4420 = WriteEA
4423 = ReadAttributes
4424 = WriteAttributes

While examining the event log I noticed that there are multiple Events generated with ID 560 for each file deletion. The first is also generated if you rename a file and contains the mask “SYNCHRONIZE”. We do not want to process these events, so we add a filter to avoid these. The third filter we add will contain the Access Mask we want to process, in our example here this will be “%%1537” which is DELETE.

Do not forget to add a “Discard” action into this rule.

A few more rules are needed in order to post process the username and domain of the user who deleted the file. If a user deletes a file on the local system, Param8 and Param9 will hold the username and domain which we need to log. If a user deletes a file on a remote system, Param11 and Param12 is important to us.

So the first rule and its actions will initialize the properties del_username and del_domain with Param8 and Param9. Please import the configuration sample to see the actions in detail.

The other two rules will be used to check if Param11 and Param 12 are empty, so not set with a useful value. If they contain any useful information, we will overwrite the del_username / del_domain property with these Params. Using this method, we can be sure to always know exactly who deleted the file.

2.3 Use File Logging to store File Deletion

The most common way to store logged operations like this is to use write a logfile. So lets create a new “Write to file” Action. Set a file path and base name of your choice, and switch the file format to “Custom”. As you can see, you can now define your own line format. In our example, I have chosen a rather simple file format, where the values are separated by comma. So you might load them into Excel later, or another other application which can load csv files.

Use the following custom line format:
%timegenerated%, %Param2%, %del_domain%\%del_username%%$CRLF%

So you will see the time the file was deleted, the deleted file (which is %Param2%) and the username and domain name. Feel free to customize the line format to your needs.

2.4 Create an Email alert

You might want to be alerted by email. But be warned, I recommend to reduce the folders and location you monitor for file deletions. Otherwise you will receive many unwanted emails. It is also possible to add custom filters to avoid emails for certain paths and filenames. Kindly filter the %Param2% property in this case.

In order to receive email alerts, create a new rule and add a new “Forward via Email” action.

Use the following subject format:
File ‘%Param2%’ deleted on Server ‘%source%’

Use the following message format:
TThe file ‘%Param2%’ was deleted on server ‘%source%’ Time:%timegenerated% Username: %del_username% Domain: %del_domain% Complete Eventmessage: %msg%

Final Thoughts

On the recycle bin: if a local file is deleted via Windows explorer, it is by default moved to the recycle bin and not actually deleted. However, the above mentioned events are still generated. This may be especially useful if you monitor e.g. a very important file via an email alert. There may be chances to undelete it even from the recycle bin.

ODBC on x64-Machines

Article created 2007-09-07 by Rainer Gerhards.

Microsoft provides integration of 32bit software into the 64 bit world. They have worked quite hard to make any differences invisible to the user and even the programmer. However, there are some subtleties that cannot be totally hidden. One of them can be experienced in the ODBC subsystem.

There are actually two ODBC subsystems in 64bit-Windows – one for the 64 bit world and one for the 32 bit world. This is necessary, because there were a number of changes that prevented 32 bit drivers from running in the 64 bit environment. In general, these two worlds are quite transparent to the user. However, there are some restrictions:

Drivers can not be shared between worlds. Most importantly, that means that many ODBC drivers (e.g. JET) are not available to 64 bit ODBC programs
Changes to the ODBC configuration made with 32 bit tools affect only the 32 bit world – and vice versa

In practice, these two restrictions limit the utility of ODBC on 64 bit Windows versions. This comes at no surprise, given that Mícrosoft has declared ODBC to be legacy some years ago and is actively promoting OLEDB (where there is a rich choice of drivers available).

When working with ODBC, please keep these restrictions on your mind. For best results, we recommend using OLEDB functionality, which is available in Adiscon products. OLEDB also offers a better performance.

“This is a step-by-step guide which describes how to Windows Update Log

How To Monitor the Windows Update Log

Article created 2007-06-13 by Florian Riedl

This Article describes you how you can monitor the Windows Update log file. This helps you to keep track of when Windows Update starts and stops working or what it does. The Windows Update log stores much more information than Windows Update writes into the EventLog.

The Article is applicable to MonitorWare Agent only.

Download MonitorWare Agent configuration file.