Identifying Anomalies in a Continuous Running Software System through Log Data Extraction

Identifying Anomalies in a Continuous Running Software System through Log Data Extraction

Siadis, J.

Gross, H.G. (mentor)

Electrical Engineering, Mathematics and Computer Science

Computer Science

Software Engineering

2013-08-09

Checking the execution behaviour of continuous running software systems is a critical task, to validate if the system is behaving as expected. In order to facilitate this process, many companies from the industry utilize log mechanisms to record events of interest and analyze the data in a post-mortem fashion. However, employing logging facilities in continuous software systems conforms to the data stream model, where the rate of log line generation is high. As a consequence, storing log lines results in enormous log files, which makes manual inspection for troubleshooting and diagnosis purposes a herculean task. Additional factors such as lack of context, noise, verbosity and multi-dimensional data further increases the difficulty to efficiently use log lines as the user has intended. As such, the challenge is to extract anomaly related information from raw log lines and raise alarms when a specified threshold value is repeatedly exceeded. This thesis takes on the challenge by proposing a solution in the form of a data extraction application and an anomaly detection application, which operate on sets of log lines sharing the same identifier and take advantage of the limited variation of log lines.

log lines
anomaly detection
request traces

http://resolver.tudelft.nl/uuid:37b78c00-b987-415e-b07a-8171827e743f

Student theses

master thesis

(c) 2013 Siadis, J.