Log Consolidator and Alerter
Adiscon today announced the immediate availability of the MonitorWare Agent 2.1 Service Pack 1 release. This is primarily a maintenance release. Continue reading “2004-07-27 MonitorWare Agent 2.1 ServicePack 1 Final Released”
Actively Monitoring Disk Free Space By Rainer Gerhards Article Date: 2004-07-22
The obvious answer is that low free space means upcoming problems, like the inability to receive mail (for mail servers) or the inability to store new files (for file servers). There are numerous obvious reasons why free space is an operations management priority. But there are also less obvious reasons: disk space shortage may be caused by a process running wild. Sometimes space consumption is the only warning indicator in such a case. Also, intruders may be the cause of low disk space conditions. For example, movie pirates often break into public servers and mis-use them as FTP servers for pirated videos. As videos are large, this can cause a sharp decrease in disk free space. In this article I primarily address the operations management needs. Obviously, the security benefits come as a side-effect. But don’t rely purely on what I am presenting here if you would like to takle the security side of disk free space. In the article, I will first convey the idea of what can be done and then I will also provide a potential solution using Adiscon’s MonitorWare Agent software.
Shortage on disk space does not (necessarily) come in an instant. Typically, free space decreases by a little every day. If left undetected, some day no space may be left at all. This is where we start at. In my point of view, a good disk space monitoring script must work with at least two thresholds:
The first level is a warning level, the second level is a real error level. In a typical setup, the warning level may not cause any big action. Typically, a notification email is sent to the administrator and that’s it. Again, in a typcial sample, the error level eventually causes more serious action. Now, the warning message may be sent to a pager email address. But a good disk space monitoring solution might also initiate some corrective action. For example, on a file server, many temporary files may fill up the disk. It may be agreed policy that such files (and eventually .bak backup files) can be automatically deleted – without asking each user. If so, a script can be started that tries to delete as many temporary files as possible, thereby freeing up disk space. In an optimal case, such a script may even delete enough space to recover from the very-low disk space condition. Ideally, it would even recover from the warning level, too. Now let’s consider that the very-low space condition triggered a pager alarm to the administrator. Poor John Admin is at the beach when his pager beeps. Too bad… Now consider he jumps off the beach and drives into his data center … just to see that the configured auto-action has already solved the issue. How would you feel in John’s place? I bet you’d be really happy and go back to the beach,wouldn’t you? I also guess you would have been even happier when the system had notified you that the low space condition was solved. So this is one more thing that we need to do within our free space monitoring: not only send an alert when things go worse, but also send an alert when the system has recovered from such a condition. Please note that the recovery case may even happen if no corrective action has been configured – just imagine a file server: a user may copy a hughe file set just to try something out. Later, he himself deletes it. Again, the low space condition is solved. Finally, a monitoring solution should only notify you once when the problem occurs and not continously (yes, I have seen solutions which do it ever and ever again…). The same goes for the “recovered” message, which obviously should only be sent once and only after a problem message has been sent first. So to sum up, a good disk free space monitoring solution must provide:
Of course, the system should be able to send different types of notifications. For example, you may want to send some of these via email while others are forwarded to a pager or a simple “net send” type notifications.
As always in life, there are many ways to implement the disk space monitor. I am using a solution based on Adiscon’s MonitorWare Agent here. This is because it is a good fit to our requested functionality and it is also easy to setup and run. MonitorWare Agent is a multi-monitoring solution. It can monitor Windows Event logs, syslog devices, databases, files … and disk space. With MonitorWare, we create a so-called disk space monitor which then is bound to a “rule set”. The disk space monitor is the part actually checking disk free space. It does this in intervals. Each time, it creates an event, which includes the free space information. That event is then passed to the rule set, where the actuall processing takes place. This is where we implement our requirements. Inside the rule set, we just need a few rules to create our scenario. Basically, we utilize MonitorWare Agent’s status variables to keep track if we have a low or a very-low space condition. With this knowledge, we check the disk space report. If it is below the thresholds and the status variable is not yet set, we create an alert (and potentially action) and set the status variable. Similarily, when free space goes up, we check if we had one of the low conditions and, if so, create another alert. We utilize MonitorWare Agent’s other action types to start the low space recovery script. Of course, I could provide you with detailled setup instructions here and also include numerous screen shots. But this article should not become a product manual… For your convenience, though, I have created a the configuration with MonitorWare Agent. You can simply download it and try it yourself. I’ve placed plenty of comments inside the rule set in that configuration. If you review the comments, you will know pretty well what I have been doing.
The MonitorWare Agent web site and free eval download
.
2004-07-22 Initial version created. 2004-10-19 Updated sample and added hyperlink to it.
Rainer Gerhards Adiscon GmbH rgerhards @ adiscon.com www.adiscon.com
The information within this paper may change without notice. Use of this information constitutes acceptance for use in an AS IS condition. There are NO warranties with regard to this information. In no event shall the author be liable for any damages whatsoever arising out of or in connection with the use or spread of this information. Any use of this information is at the user’s own risk.
Created 2004-07-15 by Tamsila-Q-Siddique.
I am using MonitorWare Agent / WinSyslog / EventReporter. How can I send the current configuration for a incident?
When working on a support incident, it is often extremely helpful to re-create a customer environment in the Adiscon lab. To aid in this process, we have added functionality to export an exact snapshot of a configuration. This is done via standard Windows registry files. Please note that when we have received your file, we are also able to make adjustments (if needed) and provide those back to you. This is a very helpful support tool.
To use it, please do the following:
You may be reluctant to send the registry file because of security reasons. We recommend you to review the contents of the registry file for security purposes with a notepad or any other text editor.
Article created 2004-07-15 by
Tamsila-Q-Siddique.
Article updated 2006-06-19 by Timm Herget.
1. You would at least need the Basic Edition of MonitorWare Agent / WinSyslog / EventReporter for this scenario.
Please Note: We are using MonitorWare Agent in this guide whereas MonitorWare Agent is
superset of WinSyslog and EventReporter. So this guide is also applicable for WinSyslog and
EventReporter.
2. When the Configuration Program client is accessed select your language – in this example, I
use English, so it might be a good idea to choose English even if that is not your preference. You
can change it any time later, but using English makes it much easier to follow this guide here.
Once done you would see a screen-shot similar to the one below:
3. Lets assume that we are interested in getting an e-mail alert in a given time period for the
following filter condition:
( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )
AND
( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )
And you also want to log the rest of the messages into a text file. The filter process will now
basically work as follow (for details see steps below):
String comparison in Filter Conditions are “Case Sensitive”! For example, if the
Source System name is “ws01” and you had written “WS01” while applying the filter, then this filter
condition would “NEVER” evaluate to True! Please double check before proceeding further!
1. In the configuration program, right click on Running Services. A menu is opened up, select
“Add Service”. Choose “Syslog Server”. Once done it will look like as below:
Once you click on the “Syslog Server” a dialog box similar to the one displayed pops up:
In this tutorial first we will create the service and then we would make the required Rule Set.
So we choose the “Create Service” option. You can opt for otherwise.
Once you have done so, a new wizard starts.
2. You can use either the default name or any other you like. I will use “My Syslog
Server” in this sample. Leave the “Use default settings” selected and
press “Next”.
3. As we have used the default settings, the wizard will immediately proceed with step 3, the
confirmation page. Press “Finish” to create the service. The wizard completes and returns
to the configuration client.
4. You will see the newly created service beneath the “Services” part of the tree
view. To check its parameters, select it:
As you can see, the service has been created with the default parameters. Please note that
there is no rule set bound to this service.
3. Define a new Rule set, right click
“Rule set”. A pop up menu will appear. Select “Add Rule set” from this
menu. On screen, it looks as follows:
4. Then, a wizard starts. Change the name of the rule to whatever name you like. We will use
“Email Alert Generation & File Logging” in this example. The screen looks as follow:
Click “Next”. A new wizard page appears.
5. Select only “Send Email”. Do not select any other options for this sample. Also, leave the
“Create a Rule for each of the following actions” setting selected. The screen looks as
follow:
6. Click “Next”. You will see a confirmation page. Click “Finish” to create
the Rule set.
7. As you can see, the new Rule set “Email Alert Generation & File Logging” is
present. We would create the “File Logging” Rule later on. Please expand the Rule Set in the tree
view until the action level of the “Send Email” Rule and select the “Send
Email” action to configure.
8. I have used factual values in the sample. In this sample I assume that the Mail Server IP
address is 192.168.0.1. The Sender and Recipient email addresses are “sender@yourdomain.com” and
“admin@yourdomain.com” respectively. Please replace these values and configure it according to your
environment.
9. Once the “Send Email” settings are configured, we will setup the filter condition. The Filter
Condition would be something like the one below:
( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )
AND
( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )
10. Click on the filter condition of the “Send Email” Rule to set up the filter condition.
11. Right click on the AND button. A pop up menu appears. Select Add Operation and then choose
the “AND” Operator. Your filter condition will look like this:
Once done, repeat the same process again. But this time Select the “OR” Operator. “AND” or “OR”
Operator are at the same level. Your filter condition will look like this:
12. Select the lower AND from the tree view and right click on the AND button. Choose “Add
Operation” from the pop up menu. Then select the OR operator. This is done to cover this part of the
filter condition “(Event ID is 500 OR 1000 OR 2000 OR 3000)”.
Right Click on the OR button. Click on the “Add Filter” from the pop up menu. Or you can use the
Add Filter Button. Select “Event Log Monitor” and then “Event ID”. This can be seen in the screen
shot below:
13. I prefer to add all four Event ID’s property filters first and later on change the
Event ID’s to the actual values in the sample. When you have added them, it should look as
follows:
14. In order to enter the actual values, select each of the four filters. A small dialog opens
at the bottom of the screen. There you enter the values you are interested in. In our sample, these
are Event ID 500, 1000, 2000, and 3000. As we are only interested in exactly these values, we do a
comparison for equality, not one of the other supported comparison modes. When you have made the
updates, you screen should look as follows:
15. Right click on the lower AND in the tree view (under which you want to add another condition
now) and click on the “Add Filter” from the pop up menu. Or you can use the Add Filter Button.
Select “General” and then “Source”.
Once the filter is added, from the “Compare Operation” combo box, select “is not equal” and
then set the value as “WS01”. When you have made the updates, you screen should look as
follows:
16. So far we have accomplished this part of the filter conditions.
( (Event ID is 500 OR 1000 OR 2000 OR 3000) ) AND ( FromHost is not equal to WS01 ) )
AND
We will work on the second part of the filter condition in the upcoming step i.e. on the
following filter:
( ( Event Source is equal to Security ) OR ( Priority is greater than 5 ) )
17. Select the lower OR from the tree view and right click on the OR button. Click on the “Add
Filter” from the pop up menu. Or you can use the Add Filter Button. Select “Event Log Monitor” and
then “Event Source”. This can be seen in the screen shot below:
Once the filter is added, from the “Compare Operation” combo box, select “is equal” and
then set the value as “Security”. When you have made the updates, you screen should look as
follows:
18. Select the lower OR from the tree view and right click on the OR button. Click on the “Add
Filter” from the pop up menu. Or you can use the Add Filter Button. Select “Syslog” and
then “Priority”. This can be seen in the screen shot below:
Once the filter is added, from the “Compare Operation” combo box, select “greater than” and
then set the value as “5”. When you have made the updates, you screen should look as
follows:
Don’t forget to save the settings by clicking the (diskette-like) “Save” button.
19. We have now selected all events that we would like to get email alerts. In order to prevent
this rule from firing too often we would enable “Minimum Wait Time”. This will make sure that (the
Syslog Facilities defined in the filter condition) in “Send Email” Rule are only forwarded once
within a specified period. Click on the Filter Conditions you would see an option called as “Global
Condition”. Select the “Minimum Wait time” and configure it. In this sample I have set the “Minimum
Wait time” to 1800 Seconds (i.e. 30 minutes). Please replace this value as you like it.
Click
here to know the difference between the Fire only if Event occurs and Minimum Wait Time.
20. We are almost done! Now we have to create a Rule for File Logging. Please note that we
are creating a “Rule” and not a “Rule Set”! The reason is that each Rule Set can have as many
Rules as you like and only one Rule Set can be associated with any service at a time (i.e My Syslog
Server in this case). Each Rule in turn can have one filter condition but as many actions as you
like. All the Rules that are part of a specific rule set are executed in a sequential manner.
In order to create a new Rule, right click on “Email Alert Generation & File Logging”
RuleSet, and select “Add Rule”. The screen looks as follow:
You can use either the default name or any other you like. I will use “File Logging” in
this sample.
21. You would see that the “File Logging” Rule has been created. If you expand the Rule in the
tree view until the action level of the “File Logging” Rule, you would notice that the
“File Logging Action” is missing. This is by default. We would create this action in the next
coming steps.
22. In order to create a “File Logging” Action, right click on the Action of the “File Logging”
Rule. A pop up menu appears. Select “Add Action.” Then opt for “Write To File”. The screen looks as
follow:
23. Then, a wizard starts. Change the name of the action to whatever name you like. We will use
“Write to File” in this example. Leave the default settings. The screen looks as
follow:
Click “Next”. You will see a confirmation page. Click “Finish” to create the
action.
24. Please select the “Write to File” action to configure.
25. The default File Path and File Base Name is “C:\temp” and “MonitorWare”. I am
using these values in this sample. You can configure it according to your environment.
Please note: If the configured directories is missing then the latest version of the
MonitorWare Agent, WinSyslog and EventReporter have the capability to create the missing
directories.
26. Leave the filter condition of “File Logging” Rule as it is. Global Conditions apply to the
rule as whole. They are automatically combined with a logical AND with the conditions in the filter
tree. The reason behind doing this is to processes all other incoming message and getting them
logged into the text file.
27. Last, save the changes if you haven’t done it before and then restart the MonitorWare /
WinSyslog or EventReporter service. This procedure completes the configuration of the Syslog
server.
MonitorWare / WinSyslog or EventReporter cannot dynamically read changed configurations. As
such,it needs to be restarted after such changes.
Article created 2004-07-12 by Tamsila-Q-Siddique.
MonitorWare Agent, WinSyslog and EventReporter enables you to apply filters to achieve your desired results. This step-by-step guide will help you through creating these filters. You can:
Please note: WinSyslog and EventReporter are subset of MonitorWare Agent i.e. MonitorWare Agent has all the features supported by WinSyslog and EventReporter (in a single place). So this step-by-step guide do apply on them as well.
Created 2004-07-08 by Tamsila-Q-Siddique.
I have MonitorWare Agent / EventReporter / WinSyslog Service on my W2K machine. And I am using Online Viewer with MSSQL as the backend. I have to reboot the machine after automatic updates for the OS or for periodical maintenance. What is the recommended order of Stopping MonitorWare Agent / EventReporter / WinSyslog Service?
This is the recommended order of stopping the services:
Please Note: MonitorWare Agent / WinSyslog / EventReporter can run under Windows NT, 2003, 2000, and XP. In addition to that MonitorWare Agent / WinSyslog / EventReporter supports Microsoft JET databases (as used by Microsoft Access), Microsoft SQL Server and MySQL. We also know of many customers who run it successfully with Oracle and Sybase as well as a variety of other systems.