Towards Detecting Patterns in Failure Logs of Large-Scale Distributed Systems

Nentawe Gurumdimma, Arshad Jhumka, Maria Liakata, Thuan Chuah, James Browne

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

10 Citations (Scopus)

Abstract

The ability to automatically detect faults or fault patterns to enhance system reliability is important for system administrators in reducing system failures. To achieve this objective, the message logs from cluster system are augmented with failure information, i.e., The raw log data is labelled. However, tagging or labelling of raw log data is very costly. In this paper, our objective is to detect failure patterns in the message logs using unlabelled data. To achieve our aim, we propose a methodology whereby a pre-processing step is first performed where redundant data is removed. A clustering algorithm is then executed on the resulting logs, and we further developed an unsupervised algorithm to detect failure patterns in the clustered log by harnessing the characteristics of these sequences. We evaluated our methodology on large production data, and results shows that, on average, an f-measure of 78% can be obtained without having data labels. The implication of our methodology is that a system administrator with little knowledge of the system can detect failure runs with reasonably high accuracy.
Original languageEnglish
Title of host publication2015 IEEE International Parallel and Distributed Processing Symposium Workshop
PublisherIEEE Explore
ISBN (Electronic)978-1-4673-7684-6
DOIs
Publication statusPublished - May 2015
Event2015 IEEE International Parallel and Distributed Processing Symposium Workshop - Hyderabad, India , Hyderabad, India
Duration: 25 May 201529 May 2015

Conference

Conference2015 IEEE International Parallel and Distributed Processing Symposium Workshop
Country/TerritoryIndia
CityHyderabad
Period25/05/1529/05/15

Fingerprint

Dive into the research topics of 'Towards Detecting Patterns in Failure Logs of Large-Scale Distributed Systems'. Together they form a unique fingerprint.

Cite this