IBM SmartCloud Analytics – Log Analysis: The Basics

IBM recently released the IBM SmartCloud – Log Analysis product, but what is it? What does it do? and how can it help you?

The aim of the product is to facilitate the analysis of log files enabling trends or issues to be quickly identified hence minimising the time taken to resolve problems and maximising availability of services. The solution can consume and index data from distributed log files to enable support teams to easily search the data from multiple sources and quickly generate data graphs from the information to identify trends and relationships.

Logged Messages Summary

Logged Messages Summary

Currently v1.1 Log Analysis WorkGroup Edition is available. There are two methods that can be used for importing log file entries, referred to as “ingesting” in the IBM documentation, either manually from a CLI or automatically using EIF events. The latter option relies on the Tivoli Log File Agent (LFA) to forward an EIF event to an “EIF Receiver” on the Log Analysis server. The LFA can either be installed on the distributed servers, locally to the monitored log files, or centrally and collect the log file entries using an SSH connection.

Irrespective of the method of receipt of the log file data, each log file entry is analysed and stored in a format to aid subsequent searching and analysis. “Insight Packs” define the required filters and rules for that initial log file analysis. Insight Packs for “DB2” and “WebSphere Application Server” are installed with the base installation. Additionally, the component Log Analysis Eclipse-plugin tooling can be used to develop custom insight packs, for example for bespoke applications.

Installation of the Log Analysis v1.1 Workgroup Edition is straight forward, although some thought will be needed when identifying which log files are to be ingested where those log files are located and the architecture for the import process. Note existing Log File Agents within the estate may be re-used if at v6.3 or greater. EIF events from earlier version may generate a Java “java.lang.StringIndexOutOfBoundsException” error in the EIF Receiver (see the full error below). Where re-using an existing LFA, a second instance can be configured specifically for the Log Analysis events.

2013-07-17 10:14:31,542 [pool-2-thread-1] ERROR – EifConsumerThread : Error while processing eifMessage:AllRecords;text='[7/17/13 10:14:22:064 BST] 0000002f 203688996     W   WIMException: com.ibm.websphere.wim.exception.InvalidUniqueNameException: CWWIM0515E  The ”UID=itnmclient,O=netcoolObjectServerRepository” entity is not in the scope of the ”defined” realm.’;RemoteHost=’UNKNOWN’;hostname=’orb-protms1′;logpath=’/opt/IBM/tivoli/tipv2/profiles/TIPProfile/logs/server1/SystemOut.log’;END
java.lang.StringIndexOutOfBoundsException
at java.lang.StringBuffer.substring(StringBuffer.java:782)
at com.ibm.tivoli.unity.eventhandlers.EifConsumerThread.jsonFromEifString(EifConsumerThread.java:170)
at com.ibm.tivoli.unity.eventhandlers.EifConsumerThread.consume(EifConsumerThread.java:85)
at com.ibm.tivoli.unity.eventhandlers.EifConsumerThread.run(EifConsumerThread.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1121)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.lang.Thread.run(Thread.java:779)

Post-installation a number of Log Analysis objects will need to be defined to enable the processing of received log files. A Log Source uniquely identifies each log file  being received based on the hostname and fully-qualified log file path, as demonstrated below.

Log Source Definition

Log Source Definition

Optionally, the Log Source topology can be defined in the file “unityServiceTopology.json”. This topology is used during the log source definition and assists the user locate the specific log files during a search (see the figure below).

Selecting a Log File

Selecting a Log File

As is standard these days users connect to the portal server using a browser, from which they can search log files, display log data in a tabular format and generate various graphical representations of the data, including bar charts, plots and bubble charts. The figure below demonstrates a bar chart generated from a db2diag log, indicating the number of distinct errors logged over a period of time.

DB2 Error Count

DB2 Error Count

What are my general thoughts?

Currently the documentation is sparse. Some effort has been made to address this through the IBM DeveloperWorks web-site, but there’s plenty of room for improvement. Administration of the product is somewhat slow through the browser interface, but I’m sure most administrators will use the CLI. The scripts used for the sample set-up provide some good examples of how to do this. Using the CLI in this fashion will greatly assist with the set-up of the multiple Log Source objects required, especially as this may be an iterative process to ensure the hostname and fully-qualified log file name exactly match those in received EIF events.

Although the currently available Insight Packs are avail restricted to DB2 and WebSphere (with Syslog added at v1.1.01?) a number of “Index” files for IBM automation products can be found on the DeveloperWorks web-site, for example TBSM and OMNIbus. However, the “Log Analysis Eclipse-plugin tooling” is key to ensuring the flexibility (and usability) of the solution by enabling the administrator to develop insight packs for any application logs desired.

Visits: 55