Principle Of Operations

Collects and converts session data from various sources

First, all session information must be collected. The different devices generates the session information for the traffic analysis at different locations and in different formats.

  • Log files from various devices, like firewalls and routers
  • SPAN ports
  • ARGUS records
  • Cisco Netflow records

The log files is parsed by either a vendor specific ASDIC parser module, or by the ASDIC generic parser module. This step converts the vendor specific textual log files to a common binary format. This makes the information available in a machine efficient format, at one single point of access and in a vendor independent format. That is, logs sources in different format from different vendors, are now all in the same format and in the same database. If a traffic pattern spans over a Cisco router, a Stonegate firewall, a Pix firewall and a Big Iron router, you no longer need to access the different systems and parse the different log formats.

SPAN ports are monitored with a specific ASDIC sniffer module, and the traffic data is converted into the database in the same format as the log data.

ARGUS records are also parsed by a specific ASDIC input data module.

Netflow via the loginput generic parser, but a native parser will be implemented in a later version.

Analysis of traffic aggregate flows

Now the database contains tons of session data in a homogeneous format. This data is easily accessible and searchable via the ASDIC web user interface. This is in itself very useful, but the real strength of ASDIC and traffic analysis is the ability to relate the individual sessions with each other to get a view over the network traffic.

This is done by the concept of aggregated sessions.

An aggregate is a meaningful collection of sessions, much like a firewall rule. See the aggregate page for a definition. When ASDIC gets new session information, it must classify it into one or more aggregate. This done without any prior knowledge of well known services, distinctions of low and high ports or so. The aggregation is done by statistics, heuristics, AI and a bit of luck. Basically, every new session seen by ASDIC is compared with every other session ever seen earlier. This is a must, and we will show you why here below.


Look at the ip datagram =>

Question: What is this?

With this information, you would guess it's a reply from a web proxy to a client. It's the reasonable answer, and might very well be the case.

But if we give it some context: => => =>

Now we can see by comparing the session with the other sessions, that it's actually not port 8080 acting as server port, but a random client port, and the true server port is port 4611.

ASDIC can by looking at the new session in the context of all the earlier sessions distinguish between same port number in different contexts.

That is, in the case above ASDIC will dismiss port 8080 as a random client port, but in some other context, ASDIC will recognize 8080 as the server port. This makes it impossible to fool ASDIC by running applications on non-standard port numbers. ASDIC have no prior knowledge of any ports, but will learn the network environment based on the actual network traffic. This is traffic analysis. Or as Arthur C. Clarke said; Any sufficiently advanced technology is indistinguishable from magic. The actual implementation is inspired by a neural network computing model.

In this example, ASDIC will assert a session aggregate into the database:* =>

In ASDIC the aggregate is marked collector because it "collects" sessions.

Note: In this school book example it may look trivial to extract the aggregate from the network traffic. In a real world example, you have more traffic involving the same hosts and the same ports in all possible permutations, and then the extraction of aggregates gets more complicated. It is a so called NP-hard problem. Due to this fact, ASDIC will not create the optimally best set of collectors, but a set good enough.

Anomaly detection

With sessions, all new sessions are ... new. This makes anomaly detection impossible. All new sessions are deviations! But with aggregates it is different. Each aggregate spans an infinite number of sessions. The total number of aggregates do not increase with the network traffic or the number of sessions, but is constant. The total number of aggregate flows increases only with network complexity. This makes it perfectly possible to detect anomalies with aggregates. After running ASDIC in your network environment, it learns the normal occurring aggregates profile and creates a baseline database. Even although this database is large, it's not approaching the infinite as a session database would, but converging to a finite number of aggregates.

Reporting things of interest

Reports can be generated in a number of ways.

  • Interactive reports via the web interface
  • Scheduled reports
  • Anomaly reports
  • Statistics reports
  • Textual reports
  • Graphic reports

The reports can be quite complex to define, and often the user run predefined reports to simplify the usage. For more detailed information, see the user manual.

Example of reports

A simple(?) way to detect possible successful cracking:

Look for all traffic from the internet to my site over the last 24 hours. Select all traffic aggregates where there has been a 100% firewall rejection of the packets in the aggregate. Use this as a indicator of bad apples, and check all traffic aggregates over the last three days from the same sources, but this time only select the traffic the firewall has permitted. If there are any matches, present it in a traffic graph over time and mail me. Do this each day.

Click to enlarge

The picture above shows a result of the described report, but in a non-firewalled environment.

To detect bad things on the internal network:

Report, if any, all traffic from our internal systems out to the internet not consistent with our firewall rule work.

To detect bad things on the internal network, deviation style:

Report, if any, all traffic from our internal systems out to the internet not consistent with out firewall rule work. Do not report active aggregates, unless this is the first occurrence of this aggregate.

Automatic inventory of new services:

Report all new tcp services found in our own network

Of course, the reports are not defined in plain English as here above, but with networks, prefixes, bitmasks and such. This is only to demonstrate the functionality. Read more about how exactly to set up reports like this in the advanced user manual found here.

Ping Research