Centralized logging in a hybrid environment (Windows/Linux)

Centralized logging in a hybrid environment (Windows/Linux)

Created 2011-03-11 by Florian Riedl

This article will describe how to setup centralized logging in a hybrid environment. Basically, we will have various major steps, that show different configuration of several clients, which forward their log data to a central loghost. There, everything will be stored into a database and processed further for alerting.

To describe the situation basically, we want all machines on the network send their log data to a central syslog server (if possible). For the central log server we take a windows machine running MonitorWare Agent (www.mwagent.com). Here we can receive syslog, monitor local log files and the Windows EventLog. Data shall be stored into a database and several email alerts shall be configured. The other steps describe the configuration of simple Windows workstations and servers, as well as Linux servers.

For TCP transmission we will use port 514 (default) for UDP and port 10514 for TCP. We want to use TCP mainly, because it ensures the transmission of the syslog messages. This is due to UDP being connectionless and thus it can occur (and will) that messages get lost.

The Client machines in this example consist of several different types of machines. We have regular Windows Workstations. Here we will use EventReporter (www.eventreporter.com). In addition to our central server, we have some other Windows Servers which will get MonitorWare Agent as well and some Linux machines which have rsyslog (www.rsyslog.com) installed. These machines will send their log messages via TCP syslog to the central server.

Additionally to these clients, we will mention some other devices and appliances (just roughly), like firewalls, switches and routers.

Step 1:

This is the first and biggest step. We will configure the central server first. The reason is simple. If this is already running, we can setup the clients and it will directly start logging everything. We assume, this is a Windows Server where MonitorWare Agent is installed. The central log server shall provide the following functionality:

  • syslog receiver with TCP (for devices that can send TCP syslog)
  • syslog receiver with UDP (for devices that can only send UDP syslog)
  • monitor the local Windows EventLog
  • monitor local textfile-logs
  • store all log messages into a database
  • send email alerts to an admin on error or critical log messages

Continue reading on Step 1

Step 2:

In step 2 we will set up the regular Windows clients. These are usually the workstations the people work on. We will use EventReporter here. It can pull all log messages from the Windows EventLog and forward them via TCP syslog. Thus the following functionality is mandatory:

  • monitor the local Windows EventLog
  • forward all log data via TCP syslog

Continue reading on Step 2

Step 3:

Now we will configure the other Windows servers. Again, we will use MonitorWare Agent because it has the most functionality. We need the following functions to be setup here:

  • monitor the local Windows EventLog
  • monitor local textfile-logs
  • forward all log data via TCP syslog

Continue reading on Step 3

Step 4:

Now we get to the Linux servers. Here we need to use a completely different product – rsyslog. For a first-time user, this might look a bit strange. The configuration we want to have here needs the following:

  • monitor local log messages
  • monitor local textfile-logs
  • forward all log data via TCP syslog

Continue reading on Step 4

Step 5:

This is rather just a note on other devices and appliances that are not yet covered. Often devices (like routers, firewalls or switches) have the possibility to send log data to a syslog server. Usually, this only works via UDP and some machines are even capable of sending logs via TCP. Since there is such a huge mass of different systems and devices, we cannot give correct steps for everything. Please refer to the user manual that came with the device or contact the manufacturer for information about how to configure the devices for sending syslog.

If you already know how to configure it, let it send it’s log messages to the central server on port 514 for UDP or (if possible) port 10514 for TCP.

Conclusion

We now have a setup that stores all the log data that machines on the network will generate to a central database for storage. Most of the clients on the network send their log data securely via TCP to the central log storage. Some machines were rather quick to set up, others needed more effort. Usually the effort rises with the amount of features that will be used. Thus we thought of this setup to be quite simple.

If you have any remarks or ideas of improvement for this guide, please let us know and send an email to info@adiscon.com.

2011-02-28 MonitorWare Agent 7.2b released

Adiscon is proud to announce the 7.2b release of MonitorWare Agent. This is a bugfixing release.

This release only consists of bugfixes:

  • Fixed an issue with percent characters in EventLog Monitor V1 and V2.
  • Fixed an RFC 2822 compatibility issue with hours which could cause problems displaying the correct time in some mailclients.
  • Corrected support for the OutputEncoding Option in ODBC and OLEDB Action related to character fields.
  • Fixed encoding detection for UTF8 encoded Syslog messages, where the BOM starts after the RFC 5424 Syslogheader.

For more details read the version history

Version 7.2b is a free download. Customers with existing 6.x keys can contact our Sales department for upgrade prices. If you have a valid Upgrade Insurance ID, you can request a free new key by sending your Upgrade Insurance ID to sales@adiscon.com. Please note that the download enables the free 30-day trial version if used without a key – so you can right now go ahead and evaluate it.

MonitorWare Agent sending to the Microsoft Message Queue

Created 2011-02-03 by Florian Riedl

With version 8.0 of MonitorWare Agent we introduced a new action called “Send MSQueue”. This action allows MonitorWare Agent Professional and Enterprise to forward the received messages to the Microsoft Message Queue. This action is also available in EventReporter Professional (v12.0) and WinSyslog Professional and Enterprise (v11.0).

To get this new functionality working, you need to do some work in advance. Basically, you need to have the Microsoft Message Queue functionality installed on both the system you want to use MonitorWare Agent on, as well as the system where you want to have the messages being forwarded to. If both systems are the same, the process is can be shortened a bit. But our assumption is, that this is not the case.

Step 1

The server which should receive the messages from the message queue needs the most attention here. We will show how to configure it with the example of a Windows 2008 server. These steps should be similar for a Windows 2003 Server as well.

Go to the Server Manager. On the left hand side, you find everything, that is necessary for the server. Go to “Features”. Then click in the right pane on “Add Features”.

Message Queue 01

You will get a list of features you can install. Right now, we are only interested in the Message Queueing feature. Mark it to be installed. You could expand the view, but for now the default options are sufficient (depending on your needs you might want to install some of the other options). When you have marked Message Queuing, click on “Next”.

Message Queue 03

You will be shown the Features to be installed. The screenshot shows only the Message-Queuing Service, since this is the only feature we want to install right now. Click on “Install”.

Message Queue 04

The feature will be installed now. When the results are shown, it should say that the installation was successful. Click on “Close” now.

Message Queue 05

In your Server-Manager you should now find Message Queuing und Features. Expand the view completely. You will see the different queues now.

Message Queue 06

Most interesting to us are the privat queues. It will hold the received messages later. But for that, we need to create a queue first. Right-click on the private queues. In the context menu go to “New” and then on “privat queue”. A window pops up, where we can define a name for it. Choose a name and click on “Ok”.

Message Queue 08

You can see the newly created queue named test on the screenshot. By expanding the view further, you will find more sub-folders like queued messages and journal messages. We will later find the messages in queued messages.

Message Queue 09

That’s it for now. The server is now ready to receive messages for the message queue.

Step 2

Now we take care of the client setup. We need to setup the message queue feature here as well. In this example we show how to do that on a Windows XP machine. The process should be similar on a Windows Vista or Windows 7 machine.

Go into the Control Panel and open “Add or Remove Programs”. Click on the “Add/Remove Windows Components” on the left side. The Windows Components Wizard will open and it will show you a list of additional programs and services. Scroll down until you find the entry “Message Queuing”.

Message Queue 10

Mark it for installation and click “Next”. The components for Message Queuing will now be installed. When it is finished, the installation wizard will tell you, that you have successfully completed the installation. Click on “Finish”. You can close the “Add or Remove Programs” window as well.

Message Queue 11

That was pretty quick. We do not have to do any extra configuration here. We just needed to install these components for the API to be available, so MonitorWare Agent can use it.

Step 3

We can now configure MonitorWare Agent to send to the Message Queue. We assume, that a basic configuration for MonitorWare Agent is already available and it is configured as a syslog receiver with a ready ruleset.

Therefore we just need to create the action. It is called “Send MSQueue”. Right click on “Actions”. A menu will open. Move the mouse to “Add Action” and the list of available actions will appear. Click on “Send MSQueue”, you will find it in the middle of the list.

Message Queue 12

The setup wizard will occur. Simply click on “Next” and then on “Finish”. You can now see the “Send MSQueue” action with its default values.

Message Queue 13

We need to change at least the “Server Computername / IP” field and the “Queuename” field. These need to be changed for the scenario to work. The rest can stay as is. Though you might want to change at least the “Queue Message Label” as this will always be the same then. You can change it, by using the properties available in MonitorWare Agent. The same goes for the field “Queue Message Body”, which can be completely customized with properties and you own content. By default it holds the message of the Syslog Message or Windows Event.

We need to change the adress field, which is on the top, to the IP of the machine we want to send to. Hostnames currently do not work yet. The “Queuename” must be set as well. This is needed for the queue that should be filled with messages. You can see on the image below, how this should look like.

Message Queue 15

You can get the path yourself by going to your queues on the server. Right click on the queue you want the path of and click on “Properties”. A window with the properties will open. In the field “Label” is the path to the queue. This should be copied and pasted into MonitorWare Agent.

Message Queue 16

Final Thoughts

This is the easiest way to set up MonitorWare Agent to work with the Microsoft Message Queue. More information on the Message Queue is available at the Microsoft website.

Please note: the MonitorWare Agent Service must be started with some account credentials that have administrative privileges on the local machine as well as on the remote server, that shall receive the messages. You might need to set this manually in the control panel for “Services”.

Massive use of memory when using TCP

Massive use of memory when using TCP

Article created 2011-02-01 by Florian Riedl.

Due to some testing we found, that in some cases MonitorWare Agent uses a lot of memory. This will happen, when MonitorWare Agent is configured as a syslog receiver and is listening to TCP. Additionally, this only occurs, when there are large message bursts, that are continously sent.

We realized it, when MonitorWare Agent used up all free memory and even the page file and didn’t stop to allocate memory blocks. First we thought this was a real bug – the memory would be allocated, but not be purged after it was used. But that was not the problem. Further testing proved, that the memory would be free again after a while.

It turned out, that the real reason was a configuration issue in MonitorWare Agent. With the default values for receiving syslog over TCP, the service would not recognize the line separators of the syslog messages, thus “thinking” it would get a single message only and thus allocating more and more memory for this “huge” message. This would happen when receiving large bursts of message and the buffer for receiving syslog messages would be full.

The solution is pretty simple, yet effective. In the configuration client you need to activate a setting for the syslog server. Just activate “Messages are separated by the following sequence” in the TCP Options. With this activated, MonitorWare Agent can properly recognize line breaks. Thus, it will only allocate memory for the real messages in the queue.

This option will be activated by default in the next release of MonitorWare Agent (8.0).

2010-12-02 MonitorWare Agent 7.2a released

Adiscon is proud to announce the 7.2a release of MonitorWare Agent. This is a bufixing release.

This release only consists of two bugfixes:

  • File Monitor
    Fixed issue an which caused message processing problems when the percent characters was within the loglines.
  • Core Engine
    Fixed string processing issues, partitially related to the SetProperty Action.

For more details read the version history

Version 7.2a is a free download. Customers with existing 6.x keys can contact our Sales department for upgrade prices. If you have a valid Upgrade Insurance ID, you can request a free new key by sending your Upgrade Insurance ID to sales@adiscon.com. Please note that the download enables the free 30-day trial version if used without a key – so you can right now go ahead and evaluate it.

MonitorWare Agent 7.2a Released (Build-IDs: Service 7.2.0.399, Client 7.2.0.1326)

MonitorWare Agent 7.2a Released

Release Date: 2010-12-02

Build-IDs: Service 7.2.0.399, Client 7.2.0.1326

Bugfixes

  • File Monitor
    Fixed issue an which caused message processing problems when the percent characters was within the loglines.
  • Core Engine
    Fixed string processing issues, partitially related to the SetProperty Action.

You can download Free Trial Version of MonitorWare Agent.

MonitorWare Agent 7.2 Released (Build-IDs: Service 7.2.0.398, Client 7.2.0.1322)

MonitorWare Agent 7.2 Released

Release Date: 2010-08-02

Build-IDs: Service 7.2.0.398, Client 7.2.0.1322

New Additions

  • Syslog Server
    – Added support for Syslog TLS (RFC 5425). It is possible to use the following modes: anonymous, x509name and x509fingerprint authentication. Note that this requires a valid server certificate as well. More details can be found in the manual.
  • Forward Syslog Action
    – Added support for Syslog TLS (RFC 5425) using your own certificate and
    keyfile, or the anonymous mode. When using the anonymous method, no client
    certifcate is needed.
  • Send Mail Action
    – Added support for STARTTLS command. This extension is required for SMTP Servers which can optionally enable encryption during communication.
  • Updated OpenSSL Components to v1.0.0 for more SSL/TLS stability

Bugfixes

  • Send Mail Action
    Fixed time issue in when “Use UTC Time” option was disabled
  • Property Engine
    Added property replacer option “replacepercent”. This option replaces all % occurrences with %%, which is needed in case that a string is reprocessed using the property engine.

You can download Free Trial Version of MonitorWare Agent.

2010-08-02 MonitorWare Agent 7.2 released

Adiscon is proud to announce the 7.2 release of MonitorWare Agent. This is a minor release including some a new feature and minor bug fixes.

As a very important enhancement, this release offers support for native and standards-compliant secure syslog transport via SSL/TLS. Based on RFC5425, MonitorWare Agent now permits sending and receiving of messages in a secure way. All RFC5425 authentication modes are supported, so messages can not only traverse the network encrypted but clients and server can also authenticate each other. Among others, this provides a reliable safeguard against man-in-the middle attacks. Note that this type of authentication is much stronger than IP-based authorization modes (as, for example, are usually found in firewalls). Of course, both can be used together for even stronger security.

The “Send Mail” Action was improved again, and now supports the STARTTLS command. This means the connection to a mailserver can be secured during transmission, if the mailserver supports it.

For more details read the version history

Version 7.2 is a free download. Customers with existing 6.x keys can contact our Sales department for upgrade prices. If you have a valid Upgrade Insurance ID, you can request a free new key by sending your Upgrade Insurance ID to sales@adiscon.com. Please note that the download enables the free 30-day trial version if used without a key – so you can right now go ahead and evaluate it.

2010-04-23 MonitorWare Agent 7.1 Final Released

MonitorWare Agent 7.1 Released

Adiscon is proud to announce the 7.1 release of MonitorWare Agent.
This release includes important new features and minor bug fixes. New features are:

  • Old-Style Format Emulation for Event Parameters – the old Windows Event Log
    system contained numbered parameters (name “ParamX” in our implementation). The new
    style implementation (e.g Windows 7, Windows Server 2008) does not support them in the same way.
    The Event Log Monitor V2 is still able to provide parameters in the “old style” format,
    what means that log analysis scripts can receive a consistent stream of data for both
    new style and old style Windows events.

  • Improved Support for NetApp Filer and similar devices
    which enables to integrate their event logs into a central repository.
    For this, EventReporter is executed on a Windows system and pulls NetApp Events via
    the NetApp event log API. Note that there are various modes for doing this, and
    all of them have some specifics in addition to the “usual” requirements found in
    the Microsoft API specification. EventReporter knows these specifics and has special
    code to work with them.

  • Support for old-style Windows Event Log Backup Files
    Current Windows releases (like Windows 2008 Server) cannot read event log backup files
    from older Windows systems utilizing the old-style event API.
    This holds also true for event log files created by third-party applications in that
    format (and there are such third-party applications). EventReporter is capable to read
    and process these files on any Windows platform, including the most recent ones.

For more details read the

version history

Version 7.1 is a free download.
Customers with existing 10.x keys can contact our Sales department for upgrade prices. If you have a valid Upgrade Insurance ID, you can request a free new key by sending your Upgrade Insurance ID to sales@adiscon.com. Please note that the download enables the free 30-day trial version if used without a key – so you can right now go ahead and evaluate it.

MonitorWare Agent 7.1 Released (Build-IDs: Service 7.1.392, Client 7.1.1307)


MonitorWare Agent 7.1 Released

Release Date: 2010-04-26

Build-IDs: Service 7.1.392, Client 7.1.1307

New Additions

 

  • EventLog Monitor V2
    – Added %ParamX% emulation Option in EventLog Monitor V2
  • EventLog Monitor V1
    – Changed Checksum verification method, so it works along with third party Eventlog applications like NETAPP Filer.
    – Implemented Fallback method to reading and process EventLog *.evt Backupfiles created on Windows XP/2003 or earlier. This workaround is needed to process these files on Windows VISTA/7/2008.
  • Send Mail Action
    – Added Switch to support old localized date header. This is needed for mail readers, that do not support UTC in date header.
  • PlaySound Action
    – Due the security service hardening in Windows, the PlaySound Action will not work on Windows VISTA/7/2008 anymore, unless you run the service in console mode.

 

Bugfixes

 

  • EventLog Monitor V2
    – Added missing check if rule processing failed.
  • EventLog Monitor V1
    – Hardened Checksum Method against various race conditions.
  • Forward Syslog Action
    – Fixed Action error handling of the Connect attempt failed.

 

You can download Free Trial Version of
MonitorWare Agent.