Can the Tivoli Log File Agent replace the TEC Logfile Adapter?

The IBM Tivoli Log File Agent is the latest incarnation of the Netcool/OMNIbus LogFile Agent. It is based on ITM autonomous agent technology and is designed to monitor text log files, Windows Event Logs and the UNIX/Linus syslog using both traditional ACP format files plus regular expressions.

This article was originally written by Simon Barnes for Message Broker 10, but advances in the Tivoli Log File Agent have demanded a review.

One glance at the diagram of current logfile monitoring options, figure 1, shows that with the advent of IBM Tivoli Monitoring (ITM) 6 and Netcool/OMNIbus there are now several choices for logfile monitoring. The alternatives now range from Netcool’s Generic Log probe, to the IBM Tivoli Monitoring Agent Builder and of course the legacy Tivoli Enterprise Console (TEC) Logfile adapter.

Despite the new offerings many customers have chosen not to move away from the tried and tested TEC Logfile adapter, not only because of the amount of effort invested in their configurations, but also because none of the options has really offered the flexibility that the existing solution’s format files already has.

However, does the latest release of the IBM Tivoli Log File Agent provide a viable alternative?

Which Log File tool?

 

Figure 1 (click to zoom)

What is the IBM Tivoli Log File Agent?

The Log File Agent is based on ITM autonomous agent technology (Autonomous agents don’t require a full ITM installation) and is designed to monitor log entries on Windows, UNIX and Linux platforms from the following sources:

  • Text log files
  • Windows Event Logs
  • UNIX/Linux Syslog

The agent uses a configuration file and a format file in the same fashion as the old TEC Logfile Adapter. Both files from the legacy Tivoli product can be reused with the Log File Agent, meaning there is no need to throw away the existing investment.

The agent collects log entries from sources identified within the configuration file. Those log entries are matched against FORMAT records defined in the format file (.fmt) and mapped to events. The FORMAT records use pattern matching based on regular expressions, although backward compatibility is maintained as legacy FORMAT record pattern matching is still supported.

The agent can forward events to both Netcool/OMNIbus (via the EIF probe) and ITM 6.2.x TEMS (for visibility through the TEP Client and analysis by defined situations).

TEC Event Adapter – Overview

Probably most people reading this article understand the concepts behind the TEC Logfile adapter but for those that don’t this section offers a small recap.

TEC Log File Adapter
Figure 2

The TEC event adapter formats a log file entry into a valid Event Integration Format (EIF) event which can then be forwarded to the TEC or OMNIbus Server. To do this it uses Format Recordsdefine within a format file named with the extension “.fmt”.

For example, consider the following log file entry for a failure of the “su” command:

Oct 15 13:41:45 kenya su: ‘su root’ failed for jerry on /dev/pts/8

The following example of a format record maps the example log file entry to an event of class Su_Failure. A format record defines an event class, pattern match and slot assignments. The pattern match includes a combination of literals and variables (%s, %t, etc), those variables can be mapped to name-value pairs, known as slots, within the event.

FORMAT Su_Failure
%t %s su: ‘su %s’ failed for %s on %s
date $1
hostname $2
from_user $4
to_user $3
on_tty $5
msg PRINTF(“su failure: %s->%s”, from_user, to_user)
source System
severity WARNING
END

In this way the message using this format is mapped to an event with the following characteristics:

Su_Failure
hostname = kenya
date = Oct 15 13:41:45
from_user = jerry
to_user = root
on_tty = /dev/pts/8
msg = su failure: jerry -> root
source = System
severity = WARNING

FORMAT records also support inheritance, where slot mappings can be inherited from a previously defined event class. The following example additionally defines the Logfile_Su event class. The Su_Failure format record “FOLLOWS”, i.e. inherits, from Logfile_Su. Mappings for the date and hostname slots do not need to be explicitly defined for the Logfile_Su format record as these are inherited from the parent event class. Functionally the mapping for the Su_Failure event class is identical to the previous example.

FORMAT Logfile_Su
%t %s su: %s*
date $1
hostname $2
msg $3
source System
severity HARMLESS
END
FORMAT Su_Failure FOLLOWS Logfile_Su
%t %s su: ‘su %s’ failed for %s on %s
from_user $4
to_user $3
on_tty $5
msg PRINTF(“su failure: %s->%s”, from_user, to_user)
severity WARNING
END

 

IBM Tivoli Log File Agent

Log File Adapter Configuration

To configure the agent you need to supply 5 pieces of information.

  • Instance Name: A unique string that identifies this instance of the agent. A single instance of the Log File Agent can monitor multiple different log sources, however, multiple instances can be configured on a single system to permit more specific and efficient filtering of the log file entries. It is suggested that the instance name indicates the application or log file that it monitors, for example webapp01 or tecad if using the legacy TEC Adapter files.
  • Conf file: Full path name to the .conf file, the configurtation file, containing the detailed configuration options. For example, /opt/IBM/LogFileAgent/webapp01.conf or /etc/Tivoli/tecad/etc/tecad_logfile.conf if using the legacy TEC Adapter files.
  • Format File: Full path name to the .fmt file, the format file, that contains the format records that you want to match in the monitored log files. For example, /opt/IBM/LogFileAgent/webapp01.fmt or /etc/Tivoli/tecad/etc/tecad_logfile.fmt if using the legacy TEC Adapter files.
  • Send EIF Events: If set to Yes (default), the agent sends EIF events to a hostname or TCP/IP address defined in the Conf file, generally a TEC or Netcool/OMNIbus server
  • Send ITM Events: If set to Yes (default is NO), log events are forwarded to the configured ITM Tivoli Enterprise Monitoring Server through normal IBM Tivoli Monitoring mechanisms.
  • Automatically initialize UNIX syslog: If set to Yes (default is “use .conf file”) the agent automatically configures the syslog facility to write a standard set of events to a pipe that the agent monitors. This option helps minimise log file housekeeping.

Figure 3, below, demonstrates the configuration screen accessed from the MTEMS application on a Windows system.


Figure 3 (click to zoom)

An agent configuration file is created in the path $ITMHOME/config (UNIX/Linux) or %ITMHOME%\TMAITM6 (Windows). The file is named  <hostname>_lo_<instance>.cfg for example itm62xsvr_lo_APPLICATIONS.cfg.

The configuration file from the previous configuration looks like this:

INSTANCE=APPLICATIONS [    SECTION=LOG_FILE_ADAPTER_CONFIGURATION [ { KLO_FORMAT_FILE=C:/IBM/LFA/app01.fmt }  { KLO_CONF_FILE=C:/IBM/LFA/app01.conf }  { KLO_AUTO_INIT_SYSLOG=No }  { KLO_SEND_ITM_EVENTS=Yes }  { KLO_SEND_EIF_EVENTS=No }  ]    SECTION=LOG_FILE_ADAPTER_GLOBAL_SETTINGS [ { KLO_PROCESS_MAX_CPU_PCT=100 }  { KLO_PROCESS_PRIORITY_CLASS=USE_CONF_FILE_VALUE }  ]]

Obviously the administration overhead for creating the configuration files increases with the number of distributed agents. One solution is to use a tool like Orb Data’s Self Service Portal (SSP) which already has integration to this new agent. In this way several agents can be managed from a web based GUI using standards and change control.

Orb Data Self Service Portal
Figure 4 (click to zoom)

Conf File
The configuration file mentioned in the agent configuration GUI is in the same format used by the TEC adapter. One point to note is that the ServerLocation and ServerPort parameters must be specified locally and use the insecure format (i.e. the IP address or hostname). For example:

ServerLocation=192.168.0.247
ServerPort=5529
BufEvtPath=/tivolilog/tecad_logfile.cache
BufferFlushRate=200
EventMaxSize=4096
PollInterval=5
LogSources=/tivolilog/tivoli.log
### FILTERS ###
Filter:Class=ADD_SLOT;
Filter:Class=Logfile_Base;
Filter:Class=Logfile_Login;

 

Netcool/OMNIbus Events

Events can be sent to Netcool/OMNIbus ObjectServer via the Tivoli EIF Probe. This probe uses a rules file to translate the received EIF event slots to fields within the Netcool/OMNIbus ObjectServer database (the table alerts.status). Verision 11 of the Tivoli EIF Probe includes a parent rules file, tivoli_eif.rules, and multiple child rules files. The child rules are disabled by default, and can optionally be enabled by removing the comment mark “#” from the relevant “include” line in the parent rules.  Default processing for events received from Tivoli Log File Agent requires the rules file tivoli_eif_tec.rules to be enabled by removing the hash from the include line within the parent rules file to read as follows:

include “tivoli_eif_tec.rules”

This will result in the  default mappings of standard event slots to OMNIbus fields, for example

  • Class -> Alert Group
  • Msg -> Summary
  • Source -> Agent

 

Netcool/OMNIbus Events
Figure 5 (click to zoom)

One common update for the default rules file is to change the mapping for the Node field. The rules shipped with version 11 of the EIF Probe populate the Node field with the content of the EIF Event slot labelled origin. Dependent on the configured format file this will often be either blank or the TCP/IP address of the system. The tivoli_eif_tec.rules logic may be updated to populate the field with the hostname as indicated below.

From if(exists($origin))
{
@Node = $origin
@NodeAlias = $origin
}
To if(exists($origin))
{
@Node = $origin
@NodeAlias = $origin
}
if(exists($hostname))
{
@Node = $hostname
}

 

 

 

 

 

 

 

 

 

It would be naive to expect the default rules to function as per a current TEC solution. One of the fundamental differences between the TEC Server and Netcool/OMNIbus solutions is their handling of duplicate detection. This needs to be considered during any migration otherwise it is likely that events will be incorrectly deduplicated by the ObjectServer. Correlation of problem and resolution events must also be considered. Such issues are considered as part of the Orb Data TEC to OMNIbus service.

ITM Events

As already stated sending events to IBM Tivoli Monitoring is also possible. This option is unlikely to be chosen within an environment migrating from TEC Server, although the agent might connect to the ITM infrastructure for management purposes. However, to minimise the resource usage, the agent may be run autonomously, i.e. with no connection to the core ITM infrastructure.

The attributes forwarded to the TEMS are a sub-set of those defined within the event. This is due to the restriction of ITM that all attributes must be predefined. By default the values of the Event Class and the msg slots are returned to the TEMS along with the Managed System Name, Log File Name and a timestamp. However, it is possible to populate some predefined custom attributes (CustomSlot1-CustomSlot10 and CustomInteger1- CustomInteger3). This is achieved from the format file by defining the attribute name after the slot definition, for example:

REGEX Base_Event
(.*): (.*)
severity HARMLESS CustomSlot1
hostname DEFAULT CustomSlot2
origin DEFAULT CustomSlot5
-tmp1 $1
-tmp2 $2
msg $2
Application APP01 CustomSlot3
EventId PRINTF(“%s:%s”, tmp1, tmp2) CustomSlot4
END
REGEX ProcessOutOfMemory FOLLOWS Base_Event
(ProcessOutOfMemory): (.*)
severity CRITICAL CustomSlot1
msg PRINTF(“Application 01 process %s is out of memory”, tmp2)
END

By defining CustomSlot standards it is possible to minimise the situation definitions. For example, if CustomSlot1 is always set to the event severity it is very easy to define a generic situation. Figure 6 demonstrates an example view containing the data collected by the Log File Agent including CustomSlot assignments as defined in the above format file extract.


Figure 6 (click to zoom)

Situations

The agent application support includes situations to alert the administrator to issues with the agent, for example configuration errors, but not to escalate log file entries. As indicated above, through the definition of standards, it would be possible to define generic situations that can be applied to all data received from the Log File Agent. Although a generic situation name is not descriptive within the situation event, the use of a short descriptive custom slot as the situation display item can indicate the nature of the failure to operators. For example a concatenation of an event class and resource may be suitable. This technique is demonstrated in the format file extract above with the EventID slot assigned to the attribute CustomSlot4. Figure 7, below, shows example situation events based on generic situation formula. The situation events are generated from the data displayed in Figure 6, above, and include the display item based on the attribute CustomSlot4.

Situation Event Console
Figure 7

I’d discourage the use of the msg attribute as the display item for performance reasons. Additionally, the use of the EifEvent attribute within situation formula should also be avoided from a performance perspective. The format file filtering should only be forwarding data that requires a user response (or is used for event clearing) and custom slots populated to align with defined standards.

Limitations

Apart from the limitations mentioned above there are a few others that you need to be aware of:

  • Configuration Options: The following conf file options are ignored:
BufferMaxSize
HostnameIsAdapterHost
LanguageID
ProcessDisablePriorityBoost
NewLogBasedOn

 

  • Configuration Distribution: One of the advantages of the Frameworks ACP profiles was that once you configured the format files on the central server there was a simple drag and drop distribution mechanism to transfer all the configuration files onto the target system. It is not as simple with ITM, although it does include a Centralized Configuration feature. (discussed in this article). The Orb Data Self Service Portal can also be used for managing such files.

 

  • Format Record Syntax: For direct TEC Adapter migrations, where legacy format records are retained, so far I’ve only unearthed one compatibility issue. The “%” sign cannot be used in isolation with the Tivoli Log File Agent, whereas that syntax is fine for the TEC Adapter. For example, a pattern match “Disk Full: %s %” works for TEC Adapters, but causes the Tivoli Log File Agent to fail initialization.

 

  • Sub Node Formats: In Sub Node mode, where multiple file pairs are read, a failure to parse one of the format files will prevent subsequent files being parsed and the sub nodes will remain off-line. This feature will not be of immediate interest to those migrating from TEC Adapter (see Additional Functionality, below, for more information on sub nodes).

 

Additional Functionality

The functionality of the Log File Agent has moved forward since the original article. The new functions may well prove useful for green field environments. Key new features include:

  • Windows 2008 Event Log: The Log File Agent has the ability to monitor application specific event logs on Windows 2008
  • Sub Nodes: This provides the ability for a single instance of the Log File Agent to use multiple configuration and format files. The files are paired, based on the base file names, and enable specific formats to be used against specific log files, for example application specific conf/fmt files can be written
  • REGEX: The legacy format syntax has now been replaced by regular expression formats, a syntax understood by the wider IT community. The legacy syntax is still consumed, but IBM recommends the REGEX format records are used for all new development.

Conclusion

So can the IBM Tivoli LogFile Agent replace the TEC Logfile adapter? I think if simple format files are used the answer is yes, but testing is imperative. The amount of work to migrate potentially thousands of format records to a different tool is so daunting that this new agent certainly looks an attractive option. In such migrations it is likely that the agent will be run in autonomous mode, forwarding events directly to Netcool/OMNIbus.

Of course there are still issues to overcome:

  • how will the distributed configuration and format files be managed?
  • how will firewalls be managed?
  • how will the agent react for complex log files?

But all of these issues can be overcome. Orb Data is offering a rapid migration service to help you understand exactly the steps that are required to move from the old TEC using Framework to this new standalone solution.

If you would like to discuss this further please contact Orb Data on +44 (0) 1628 550450 or email info@orb-data.com .

To download the product search for IBM Tivoli Monitoring V6.2.3.2 Log File Agent Multiplatform or use the Product code CI8WDEN.