ASDIC

Benchmark


ASDIC is designed with performance in mind and is entirely written in the C programming language.

Benchmark test setup

  • Firewall log file 80 412 720 entries, 20GB compressed to 1.3GB (gzip)
  • Default log file parse template /etc/opt/asdic/loginput.dat with all redundant entries left in
  • 44% av log entries parsable, 56% other data
  • Q9450 Intel quad core processor at 2.66GHz
  • Ubuntu 8.04 GNU/Linux distribution
  • ASDIC version 3.0.0
  • SATA hard drives
  • 8GB RAM memory
  • history database hash size 50 000 000

Test command

# time zcat logfile.gz | loginput -s /etc/opt/asdic/loginput.dat

Result

  • 477 seconds real time
  • Average system load 2.8
  • 40MB log data per second
  • ~170 000 log entries per second in total
  • ~74 000 entries per seconds of parsable data

Notes

  • Redundant (i.e. non used) entries in parse template file makes parsing slower
  • Non parsable entries in log file makes parsing slower
  • Running wc(1) on the same log data takes 1001s real time.
  • Running grep(1) for an IP address 511s real time.
  • Ubuntu 8.10 and 9.04 runs about 20% slower due to unsound cpu affinity in newer kernels.

Ping Research

ASDIC