ASDIC

Loginput


NAME

loginput - generic log parser

SYNOPSIS

/opt/asdic/bin/loginput [-tf] [-s loginput.dat] [file]

DESCRIPTION

Parses generic text log file for traffic information. Uses a configuration file loginput.dat with log templates.

OPTIONS

-f
Run as filter. Output everything not matching template to standard output.
-t
Run in test mode. Print grammar before and after parsing. Print numbers of matches of rules.
-s file
Use file for definition file instead of default loginput.dat
file
File to parse. If no file name, standard input is used.

CONFIGURATION

The loginput.dat file specifies the log file template. It uses variables to parse the specific fields in the log entries. The configuration file should really not be treated as a text file, but a binary file. Leading and trailing spaces are significant. Edit with care and never cut and paste.

$TS
Time Stamp
$SA
Source ip Address
$SP
Source Port
$DA
Destination ip Address
$DP
Destination Port
$TP
Transport Protocol (tcp/udp/icmp)
$TA
Type of Access (accept/deny)
$NP
Number of packets (optional, default 1).
$FA
Firewall ip Address (optional, for rule tracking)
$FR
Firewall Rule number (optional, for rule tracking)
#
A sequence of digits of at least length 1.
*
A sequence of any length of any characters except for the character following the asterisk. E.g. "* " matches non-whitespace, ";*;" matches anything between the semicolons, except semicolon itself.
(space)
Any number of white-spaces of at least length 1.

To parse the log file format

Aug 7 00:01:03 cisco.ping.se 2840090: Aug 7 00:01:03.446 MET-DST: %SEC-6-IPACCESSLOGP: list my-acl denied udp 192.168.196.50(1127) -> 10.0.179.194(161), 1 packet

use the following loginput.dat 

[syntax]
$TS cisco.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)

You can use multiple lines with templates after the syntax directive, and the first match is used. Also note that the template matches when it ends, even if there are more characters in the input line. Therefore there is no need to end with "# packet*" in this example. Also note that trailing spaces are significant.

The $TA variable matches a multitude of access tokens, like "accept", "permitted", "deny" etc. If not matching a particular token, you can override the default $TA definition in the loginput.dat definition file; 

[accept]
accept*
granted

The first wild card line will match "accept", "accepted" etc. This is not a regular expression. The optional asterisk can only be the at the last position. The second line will match only "granted". Case is not significant. Deny tokens can be defined in the same way under the heading [deny]. For best performance, keep the defined tokens sorted so the most frequent token is defined first.

Timestamp can be defined in the same manner, in strptime(3C) syntax. Multiple definitions can be used used. Year is optional. If not specified, the "closest" year is default, e.g. current year, or if log entry is from December and current month is January, last year will be used as default. Note that %Z can not be used for time zone.

Default timestamp definition (e.g. "Oct 23 12:34:56"); 

[timestamp]
%b %d %T

Time zone, if other then TZ environment variable. Uses the zoneinfo syntax. See /usr/share/zoneinfo for your time zone or leave out for system default. 

[timezone]
UTC

If several different log templates all start with the same beginning, you can organize the template in a more efficient way; 

[syntax]
$TS cisco.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)
$TS gw.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)

Is equivalent with 

[syntax]
$TS 
        cisco.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)
        gw.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)

The indentation must be one TAB character, and this structure can be used in multiple levels. Make sure you keep the white space between $TS and "cisco.ping.se", preferably directly after the $TS before the new line. For performance, order the templates so the most frequent one comes first. This can be verified with the -t test option.

If the log input does not contain a defined order of the fields (like Firewall-1), a similar construction can be used. 

[syntax]
$TS $TA 
        src: $SA;$<<
        dst: $DA;$<<
        s_port: $SP;$<<
        service: $DP;$<<
         $<<
        *: *;$<<

The $<< token instructs the parser not to advance to the next state. When this is used, the parser accept state is when all the input is consumed, or when a rule not ending with $<< is encountered. Note the last two lines; the first of them skips white space, and the last skips non-recognized tokens.

The $<> token instructs the parser to unconditionally discard the entire log entry without inserting any data into ASDIC. This can be used to silent warnings about unparsable log entries, or simple to throw away unwanted log entries.

Everything before the first section will be treated as a comment. 

Parse cisco style log entries from our core router

[syntax]
$TS cisco.ping.se #: * # #:#:#.# MET-DST: %SEC-6-IPACCESSLOGP: list * $TA $TP $SA($SP) -> $SA($DP)

EXAMPLES

The examples above are more to explain the mechanisms, then for real world usage. Here below are some more useful examples. Please note that cut and paste may not work because tabs and trailing spaces might be lost. 

Parse cisco log entries. Handle icmp entries also.

[syntax]
$TS *%SEC-6-IPACCESS* list # $TA $TP 
        $SA($SP) -> $DA($DP), $NP packet
        $SA -> $DA ($SP/$DP), $NP packet

NOTES

loginput do not use ordinary regular expressions of performance reasons.

Ping Research

ASDIC