IHE and the syslog message size

Written 2005-08-04 by Rainer Gerhards.

This is work in progress, expect updates to this page!

This is not yet a real article, but more a lab notebook. I had an interesting discussion on IHE and syslog. The issue with that is that IHE defines log records of up to 32K, while syslog only allows records of up to 1k - at least in current standards. Thankfully, many syslog implementations do not take this limit as fixed and ignore the standard in that regard. Also, the upcoming new standard allows for larger messages, so this issue somewhat disappears.

Anyhow... This time, I was curious enough to do a lab session on what needs to be done to make the receivers "IHE-aware". I am noting down the results on this page and will update it whenever I have some additional results. I will also update it when I have some more ideas on how to make IHE really work with current syslog.

I used two simple perl scripts as the syslog clients/senders. Each of them tries to send a ca. 32K test message (with nonsense data) to a syslog server. One does it via udp, the other via tcp.

Results

Notes:

• WinSyslog and MonitorWare Agent have no inherent message size limitation and accept whatever the IP stack forwards to them. So test were conducted without any special setup.
• Rsyslogd has a hardcoded size limitation, which is easily modifyable. In the table, rsyslogd[33k] means rsyslogd compiled with support for 33k message size, rsyslogd[1k] means compiled with 1k message size (the default).
• The Windows workstation used in the table above was Windows XP SP2, Build 2600
• most test were conducted using local area networking, some with WLAN (as an additional checkpoint)
• all test were performed at least three time, more often if there were some inconsistencies between the test results. Additional tests were also done in the case where only 7 out of 8 message segements were received. All results are clearly reproducable.

(Preliminary) Conclusion

UDP-based syslog is unable to support more than 4k of message size, even if the receiver application imposes no limit. RFC 3164 based solutions might be extended to glue packets together, but this mode is not well-defined. Different operating systems seem to come to different results in the amount of data that can be transmitted. An approach might be to send large messages in multiple chunks and have the receiver glue them together. However, that would require specification.

TCP-based syslog provides full support for 32k message sizes (and beyond), but is not currently standardized.

RFC 3195 syslog would support larger message sizes in theory. As was found out, one can argue that no strict upper message limit size is specified (though I think this is more an omission in the RFC, but we might use that to our advantage). However, there is currently no known implementation supporting messages of larger than 1k. Liblogging seems to be modifiable to support it with moderate effort. However, this needs to be proven, especially if a "clean" solution is desired (that would lead to full implementation of BEEP framing and segmentation).

Revision History

• 2005-08-04 * Rainer Gerhards * initial version created

Copyright

Copyright (c) 2005 Rainer Gerhards and Adiscon.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at http://www.gnu.org/copyleft/fdl.html.