adisconteam, Author at MonitorWare Agent

Actively Monitoring Disk Free Space

Actively Monitoring Disk Free Space By Rainer Gerhards Article Date: 2004-07-22

Why care about disk free space?

The obvious answer is that low free space means upcoming problems, like the inability to receive mail (for mail servers) or the inability to store new files (for file servers). There are numerous obvious reasons why free space is an operations management priority. But there are also less obvious reasons: disk space shortage may be caused by a process running wild. Sometimes space consumption is the only warning indicator in such a case. Also, intruders may be the cause of low disk space conditions. For example, movie pirates often break into public servers and mis-use them as FTP servers for pirated videos. As videos are large, this can cause a sharp decrease in disk free space. In this article I primarily address the operations management needs. Obviously, the security benefits come as a side-effect. But don’t rely purely on what I am presenting here if you would like to takle the security side of disk free space. In the article, I will first convey the idea of what can be done and then I will also provide a potential solution using Adiscon’s MonitorWare Agent software.

The Idea

Shortage on disk space does not (necessarily) come in an instant. Typically, free space decreases by a little every day. If left undetected, some day no space may be left at all. This is where we start at. In my point of view, a good disk space monitoring script must work with at least two thresholds:

disk space is low, but still acceptable
disk space is too low, problems will occur very soon (or already exist)

The first level is a warning level, the second level is a real error level. In a typical setup, the warning level may not cause any big action. Typically, a notification email is sent to the administrator and that’s it. Again, in a typcial sample, the error level eventually causes more serious action. Now, the warning message may be sent to a pager email address. But a good disk space monitoring solution might also initiate some corrective action. For example, on a file server, many temporary files may fill up the disk. It may be agreed policy that such files (and eventually .bak backup files) can be automatically deleted – without asking each user. If so, a script can be started that tries to delete as many temporary files as possible, thereby freeing up disk space. In an optimal case, such a script may even delete enough space to recover from the very-low disk space condition. Ideally, it would even recover from the warning level, too. Now let’s consider that the very-low space condition triggered a pager alarm to the administrator. Poor John Admin is at the beach when his pager beeps. Too bad… Now consider he jumps off the beach and drives into his data center … just to see that the configured auto-action has already solved the issue. How would you feel in John’s place? I bet you’d be really happy and go back to the beach,wouldn’t you? I also guess you would have been even happier when the system had notified you that the low space condition was solved. So this is one more thing that we need to do within our free space monitoring: not only send an alert when things go worse, but also send an alert when the system has recovered from such a condition. Please note that the recovery case may even happen if no corrective action has been configured – just imagine a file server: a user may copy a hughe file set just to try something out. Later, he himself deletes it. Again, the low space condition is solved. Finally, a monitoring solution should only notify you once when the problem occurs and not continously (yes, I have seen solutions which do it ever and ever again…). The same goes for the “recovered” message, which obviously should only be sent once and only after a problem message has been sent first. So to sum up, a good disk free space monitoring solution must provide:

at least two thresholds for disk space shortages
notifications that only occur ones these thresholds are crossed
optionally automatically-triggered corrective actions
notifications when the shortage conditions have been triggered

Of course, the system should be able to send different types of notifications. For example, you may want to send some of these via email while others are forwarded to a pager or a simple “net send” type notifications.

A potential Solution

As always in life, there are many ways to implement the disk space monitor. I am using a solution based on Adiscon’s MonitorWare Agent here. This is because it is a good fit to our requested functionality and it is also easy to setup and run. MonitorWare Agent is a multi-monitoring solution. It can monitor Windows Event logs, syslog devices, databases, files … and disk space. With MonitorWare, we create a so-called disk space monitor which then is bound to a “rule set”. The disk space monitor is the part actually checking disk free space. It does this in intervals. Each time, it creates an event, which includes the free space information. That event is then passed to the rule set, where the actuall processing takes place. This is where we implement our requirements. Inside the rule set, we just need a few rules to create our scenario. Basically, we utilize MonitorWare Agent’s status variables to keep track if we have a low or a very-low space condition. With this knowledge, we check the disk space report. If it is below the thresholds and the status variable is not yet set, we create an alert (and potentially action) and set the status variable. Similarily, when free space goes up, we check if we had one of the low conditions and, if so, create another alert. We utilize MonitorWare Agent’s other action types to start the low space recovery script. Of course, I could provide you with detailled setup instructions here and also include numerous screen shots. But this article should not become a product manual… For your convenience, though, I have created a the configuration with MonitorWare Agent. You can simply download it and try it yourself. I’ve placed plenty of comments inside the rule set in that configuration. If you review the comments, you will know pretty well what I have been doing.

Related Software

The MonitorWare Agent web site and free eval download

Revision History

2004-07-22 Initial version created. 2004-10-19 Updated sample and added hyperlink to it.

Author’s Address

Rainer Gerhards Adiscon GmbH rgerhards @ adiscon.com www.adiscon.com

Disclaimer

The information within this paper may change without notice. Use of this information constitutes acceptance for use in an AS IS condition. There are NO warranties with regard to this information. In no event shall the author be liable for any damages whatsoever arising out of or in connection with the use or spread of this information. Any use of this information is at the user’s own risk.

How can I send my configuration in a support case?

Created 2004-07-15 by Tamsila-Q-Siddique.

I am using MonitorWare Agent / WinSyslog / EventReporter. How can I send the current configuration for a incident?

When working on a support incident, it is often extremely helpful to re-create a customer environment in the Adiscon lab. To aid in this process, we have added functionality to export an exact snapshot of a configuration. This is done via standard Windows registry files. Please note that when we have received your file, we are also able to make adjustments (if needed) and provide those back to you. This is a very helpful support tool.

To use it, please do the following:

Go to “Computer Menu”
Choose “Export Settings to Registry-File” be sure NOT to select a binary format – they are only for special purposes. You can also NOT review binary files for security-relevant data.
Save this registry file.

You may be reluctant to send the registry file because of security reasons. We recommend you to review the contents of the registry file for security purposes with a notepad or any other text editor.

How do I Add filters for MonitorWare Agent, WinSyslog and EventReporter?

Article created 2004-07-15 by
Tamsila-Q-Siddique.

Article updated 2006-06-19 by Timm Herget.

1. You would at least need the Basic Edition of MonitorWare Agent / WinSyslog / EventReporter for this scenario.

Please Note: We are using MonitorWare Agent in this guide whereas MonitorWare Agent is
superset of WinSyslog and EventReporter. So this guide is also applicable for WinSyslog and
EventReporter.

2. When the Configuration Program client is accessed select your language – in this example, I
use English, so it might be a good idea to choose English even if that is not your preference. You
can change it any time later, but using English makes it much easier to follow this guide here.
Once done you would see a screen-shot similar to the one below:

3. Lets assume that we are interested in getting an e-mail alert in a given time period for the
following filter condition:

( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )

AND

( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )

And you also want to log the rest of the messages into a text file. The filter process will now
basically work as follow (for details see steps below):

Rule 1: Finds the Filter condition stated above and makes sure it is only reported
once within a given period. Later on when the required filter condition is evaluated to true,
an e-mail alert is generated.
Rule 2: Processes all other incoming message and log them into text file.

Important note about Filter Condition

String comparison in Filter Conditions are “Case Sensitive”! For example, if the
Source System name is “ws01” and you had written “WS01” while applying the filter, then this filter
condition would “NEVER” evaluate to True! Please double check before proceeding further!

Step 1 – Create a Syslog Server

1. In the configuration program, right click on Running Services. A menu is opened up, select
“Add Service”. Choose “Syslog Server”. Once done it will look like as below:

Once you click on the “Syslog Server” a dialog box similar to the one displayed pops up:

In this tutorial first we will create the service and then we would make the required Rule Set.
So we choose the “Create Service” option. You can opt for otherwise.

Once you have done so, a new wizard starts.

2. You can use either the default name or any other you like. I will use “My Syslog
Server” in this sample. Leave the “Use default settings” selected and
press “Next”.

3. As we have used the default settings, the wizard will immediately proceed with step 3, the
confirmation page. Press “Finish” to create the service. The wizard completes and returns
to the configuration client.

4. You will see the newly created service beneath the “Services” part of the tree
view. To check its parameters, select it:

As you can see, the service has been created with the default parameters. Please note that
there is no rule set bound to this service.

Step 2 – Create a Rule Set for Email Alert Generation and File Logging

3. Define a new Rule set, right click
“Rule set”. A pop up menu will appear. Select “Add Rule set” from this
menu. On screen, it looks as follows:

4. Then, a wizard starts. Change the name of the rule to whatever name you like. We will use
“Email Alert Generation & File Logging” in this example. The screen looks as follow:

Click “Next”. A new wizard page appears.

5. Select only “Send Email”. Do not select any other options for this sample. Also, leave the
“Create a Rule for each of the following actions” setting selected. The screen looks as
follow:

6. Click “Next”. You will see a confirmation page. Click “Finish” to create
the Rule set.

7. As you can see, the new Rule set “Email Alert Generation & File Logging” is
present. We would create the “File Logging” Rule later on. Please expand the Rule Set in the tree
view until the action level of the “Send Email” Rule and select the “Send
Email” action to configure.

8. I have used factual values in the sample. In this sample I assume that the Mail Server IP
address is 192.168.0.1. The Sender and Recipient email addresses are “sender@yourdomain.com” and
“admin@yourdomain.com” respectively. Please replace these values and configure it according to your
environment.

9. Once the “Send Email” settings are configured, we will setup the filter condition. The Filter
Condition would be something like the one below:

( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )

AND

( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )

10. Click on the filter condition of the “Send Email” Rule to set up the filter condition.

11. Right click on the AND button. A pop up menu appears. Select Add Operation and then choose
the “AND” Operator. Your filter condition will look like this:

Once done, repeat the same process again. But this time Select the “OR” Operator. “AND” or “OR”
Operator are at the same level. Your filter condition will look like this:

12. Select the lower AND from the tree view and right click on the AND button. Choose “Add
Operation” from the pop up menu. Then select the OR operator. This is done to cover this part of the
filter condition “(Event ID is 500 OR 1000 OR 2000 OR 3000)”.

Right Click on the OR button. Click on the “Add Filter” from the pop up menu. Or you can use the
Add Filter Button. Select “Event Log Monitor” and then “Event ID”. This can be seen in the screen
shot below:

13. I prefer to add all four Event ID’s property filters first and later on change the
Event ID’s to the actual values in the sample. When you have added them, it should look as
follows:

14. In order to enter the actual values, select each of the four filters. A small dialog opens
at the bottom of the screen. There you enter the values you are interested in. In our sample, these
are Event ID 500, 1000, 2000, and 3000. As we are only interested in exactly these values, we do a
comparison for equality, not one of the other supported comparison modes. When you have made the
updates, you screen should look as follows:

15. Right click on the lower AND in the tree view (under which you want to add another condition
now) and click on the “Add Filter” from the pop up menu. Or you can use the Add Filter Button.
Select “General” and then “Source”.

Once the filter is added, from the “Compare Operation” combo box, select “is not equal” and
then set the value as “WS01”. When you have made the updates, you screen should look as
follows:

16. So far we have accomplished this part of the filter conditions.

( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )

AND

We will work on the second part of the filter condition in the upcoming step i.e. on the
following filter:

( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )

17. Select the lower OR from the tree view and right click on the OR button. Click on the “Add
Filter” from the pop up menu. Or you can use the Add Filter Button. Select “Event Log Monitor” and
then “Event Source”. This can be seen in the screen shot below:

Once the filter is added, from the “Compare Operation” combo box, select “is equal” and
then set the value as “Security”. When you have made the updates, you screen should look as
follows:

18. Select the lower OR from the tree view and right click on the OR button. Click on the “Add
Filter” from the pop up menu. Or you can use the Add Filter Button. Select “Syslog” and
then “Priority”. This can be seen in the screen shot below:

Once the filter is added, from the “Compare Operation” combo box, select “greater than” and
then set the value as “5”. When you have made the updates, you screen should look as
follows:

Don’t forget to save the settings by clicking the (diskette-like) “Save” button.

19. We have now selected all events that we would like to get email alerts. In order to prevent
this rule from firing too often we would enable “Minimum Wait Time”. This will make sure that (the
Syslog Facilities defined in the filter condition) in “Send Email” Rule are only forwarded once
within a specified period. Click on the Filter Conditions you would see an option called as “Global
Condition”. Select the “Minimum Wait time” and configure it. In this sample I have set the “Minimum
Wait time” to 1800 Seconds (i.e. 30 minutes). Please replace this value as you like it.

Click
here to know the difference between the Fire only if Event occurs and Minimum Wait Time.

20. We are almost done! Now we have to create a Rule for File Logging. Please note that we
are creating a “Rule” and not a “Rule Set”! The reason is that each Rule Set can have as many
Rules as you like and only one Rule Set can be associated with any service at a time (i.e My Syslog
Server in this case). Each Rule in turn can have one filter condition but as many actions as you
like. All the Rules that are part of a specific rule set are executed in a sequential manner.

In order to create a new Rule, right click on “Email Alert Generation & File Logging”
RuleSet, and select “Add Rule”. The screen looks as follow:

You can use either the default name or any other you like. I will use “File Logging” in
this sample.

21. You would see that the “File Logging” Rule has been created. If you expand the Rule in the
tree view until the action level of the “File Logging” Rule, you would notice that the
“File Logging Action” is missing. This is by default. We would create this action in the next
coming steps.

22. In order to create a “File Logging” Action, right click on the Action of the “File Logging”
Rule. A pop up menu appears. Select “Add Action.” Then opt for “Write To File”. The screen looks as
follow:

23. Then, a wizard starts. Change the name of the action to whatever name you like. We will use
“Write to File” in this example. Leave the default settings. The screen looks as
follow:

Click “Next”. You will see a confirmation page. Click “Finish” to create the
action.

24. Please select the “Write to File” action to configure.

25. The default File Path and File Base Name is “C:\temp” and “MonitorWare”. I am
using these values in this sample. You can configure it according to your environment.

Please note: If the configured directories is missing then the latest version of the
MonitorWare Agent, WinSyslog and EventReporter have the capability to create the missing
directories.

26. Leave the filter condition of “File Logging” Rule as it is. Global Conditions apply to the
rule as whole. They are automatically combined with a logical AND with the conditions in the filter
tree. The reason behind doing this is to processes all other incoming message and getting them
logged into the text file.

27. Last, save the changes if you haven’t done it before and then restart the MonitorWare /
WinSyslog or EventReporter service. This procedure completes the configuration of the Syslog
server.

MonitorWare / WinSyslog or EventReporter cannot dynamically read changed configurations. As
such,it needs to be restarted after such changes.

How do I apply filters in MonitorWare Agent, WinSyslog and EventReporter?

Article created 2004-07-12 by Tamsila-Q-Siddique.

MonitorWare Agent, WinSyslog and EventReporter enables you to apply filters to achieve your desired results. This step-by-step guide will help you through creating these filters. You can:

Please note: WinSyslog and EventReporter are subset of MonitorWare Agent i.e. MonitorWare Agent has all the features supported by WinSyslog and EventReporter (in a single place). So this step-by-step guide do apply on them as well.

What is the recommended order of Stopping MonitorWare Agent / EventReporter / WinSyslog Service?

Created 2004-07-08 by Tamsila-Q-Siddique.

I have MonitorWare Agent / EventReporter / WinSyslog Service on my W2K machine. And I am using Online Viewer with MSSQL as the backend. I have to reboot the machine after automatic updates for the OS or for periodical maintenance. What is the recommended order of Stopping MonitorWare Agent / EventReporter / WinSyslog Service?

This is the recommended order of stopping the services:

Stop IISadmin
Stop MonitorWare Agent / EventReporter / WinSyslog Service
Stop MSSQL Server

Please Note: MonitorWare Agent / WinSyslog / EventReporter can run under Windows NT, 2003, 2000, and XP. In addition to that MonitorWare Agent / WinSyslog / EventReporter supports Microsoft JET databases (as used by Microsoft Access), Microsoft SQL Server and MySQL. We also know of many customers who run it successfully with Oracle and Sybase as well as a variety of other systems.

How can I make Event ID part of the actual Syslog message while forwarding to a Syslog Server?

Created 2004-06-24 by Tamsila-Q-Siddique.

We are using MonitorWare Agent / EventReporter to forward Windows Event logs to a Syslog Server. The resulting syslog message doesn’t have the Event IDs in them. How can we make Event ID part of the actual Syslog message?

One of the proposed solution would be to forward the Event Log messages using SETP Server. The resulting message would have the Event IDs in them. Click here to know the difference between SETP and Syslog!

But there are other ways to include the Event ID even without using SETP (which is obviously not an option if you would like to send to a non-Adiscon backend). So you can do one of the following:

Use XML Format – This is the best recommended option. With XML format, you get everything about this event and you get it in a well-structured way. It includes all of the properties described in our Event Properties reference. To enable XML format, simply check “Use XML to Report” in the “Forward Syslog” Action.
Use Custom Format – In the “Forward Syslog” action, you can specify your own custom format in the “Message Format” text box. By default it is set to %msg%, but you can include whatever you like. Use the “Insert” link to do this (or simply type it)! Be sure to read the Property Replacer” documentation to see the full power. This option is a good one, especially if you intend to parse the data… because *you* can exactly specify what you would like to see.
Use MoniLog Format – This is our former legacy format. It includes a bunch of useful information, but it has a number of anomalies, which might hit you in few cases when parsing. We do not recommend it, but if you would like to use it, you can select the “Insert” link in the “Forward Syslog” action properties. Then, select “Replace with MoniLog Format”. It will generate a custom format of the type given below. Again, we do not recommend this, but it is a way.## %severity% %timereported:::uxTimeStamp%: %source%/%sourceproc% (%id%) – “%msg%” ##
Change Event Log Monitor Settings – You could also change the Event Log Monitor itself to generate the legacy format. Then, you do not need to change the “Forward Syslog” action’s settings. The big drawback is that now the Event Log Monitor does emit an old format, which is not meant to be processed by any other MonitorWare product. If you just use the product as a back-end for your own front-end, this is not an issue. Anyhow, we still recommend to go for approach #3 instead of this. If you absolutely want to do it this way, this is how it is done:
Go to the Event Log Monitor properties. Click on the “Advanced Options” button. Check the “Use Legacy Format” checkbox. This will enable some other checkboxes. Review the options to see which of these you want.

We have provided the options at hand. We *strongly* recommend to go for either option 1 or 2. If you choose option 3 or 4, you can receive a parsing error from time to time. However this has been solved after introducing the newer formats.

As a general hint, you may want to take into account that Windows Event Log messages can become rather lenghty. They often go over the syslog RFC size of 1024 bytes. If you run a non-Adiscon Syslog Server, you need to ensure it can receive such large messages, because otherwise some information might be missing (with option 2, you can customize what you would like to be missing in such cases – by limiting the size of %msg% via the property replacer).

System Requirements for Monitoring NetWare Files

Created on 2003-08-08 by Rainer Gerhards.
Updated on 2004-06-16 by Tamsila-Q-Siddique.

MonitorWare Agent needs to access files on NetWare via an UNC share. It is known that some versions of the Novell and/or Microsoft software have some issues with services accessing files on a UNC share on NetWare.

Microsoft acknowledges that there is a problem in Windows 2000 without any service pack. For more information, go to:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;250502

However, we had the same problem with Windows 2000 SP1. It was resolved when SP3 was installed.

So as a general advice, we strongly recommend using the latest Microsoft service pack available for your operating system. If you are using the Novell Client, you should also use the most recent one.

Why does the File Monitor Service experience difficulties when accessing files located on a NetWare Server?

Created 2004-06-16 by Tamsila-Q-Siddique.

I am attempting to watch files on a NetWare Server. On my W2K machine MonitorWare Agent can monitor files on itself wonderfully but it will not monitor files on the NetWare Server. We are receiving your error code 1707 when attempting to connect via your connection menu item. What to do?

This is because our MonitorWare Line of Products (e.g. MonitorWare Agent) are installed to be running under the “Windows Local System Account”. This account is restricted from network I/O by Windows design.

We have also seen that the NetWare Client under Windows 2000 seems to have big issues with services. Click here to know the recommended System Requirements for Monitoring NetWare Files.

Using the Microsoft Client – and not the Novell one – will most likely solve any issues. You can verify if it is a client issue, by running the MonitorWare Agent in the fore ground, and not as a service. This was specifically added as a work-around for Novell related issues (we have really, really pinpointed that this is a Novell / Microsoft issue and not our software – lot’s of other services are experiencing similar problems). The Novell issue is only with services – as soon as you run the MonitorWare Agent in a DOS box in an interactive session, all troubles disappear.

Using a UNC share to monitor a log file can be used as a work around as well.

Why do I get “Type Mismatch” or “Page Not Found” Error when using the Online Web Access Viewer?

Created 2004-06-15 by Tamsila-Q-Siddique

I have verified all the settings in the “ConfigSettings.asp” page and it looks fine. All Permissions are granted. But when I access the Online Web Access Viewer the “Type Mismatch” or “Page Not Found” error is displayed. What to do?

You have probably “Disabled Session State”. To run the Winsyslog Web Access / MonitorWare Web Access you must enable the Session State. The reason has been documented in the Microsoft Knowledge Base Article – 242425.

Why does the Port Probe Service Fails?

Created 2004-06-15 by Tamsila-Q-Siddique

I have configured a PortProbe Service to check for activity of the SMTP Service on our mail server. MonitorWare Agent has full Internet access and I am not using any proxy servers or DNS-aliases for the mailserver. The PortProbe service is running but it does not execute the action configured (i.e. when target port can’t be connected). What am I doing wrong?

If the PortProbe Service is running but the action configured is not executed then it can be one of the following reasons:

On the PortProbe Service properties window, there is an option called “Generate an event if PortProbe was successfull”. Kindly uncheck this option, and it will only generate an event if the target port can not be connected.
Your mail server is protected by a Firewall.
You may have a personal Firewall running that is blocking MonitorWare Agent to reach the mail server.