Unsupervised anomaly detection in large databases using Bayesian networks
Loading...
Date
2008
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
TAYLOR & FRANCIS INC
Abstract
Today, there has been a massive proliferation of huge databases storing valuable information. The opportunities of an effective use of these new data sources are enormous; however the huge size and dimensionality of current large databases calls for new ideas to scale up current statistical and computational approaches. This article presents an application Of artificial intelligence technology to the problem of automatic detection of candidate anomalous records in a large database. IMP build our approach with three main goats in mind: 1) an effective detection of the records that are potentially anomalous; 2) a suitable selection of the subset of attributes that. explains what makes a record anomalous; and. 3) an efficient implementation that allows us to scale the approach to large databases. Our algorithm called Boyesian network anomaly detector (BNAD), uses the joint probability density junction (pdf) provided by a Bayesian network (BN) to achieve these goals. By using appropriate data structures, advanced caching techniques, the flexibility of Gaussian mixture mod els, Find the efficiency of BNs to model joint pdfs, BNAD manages to efficiently learn a suitable BV from a large dataset. We test BNAD using synthetic and real databases, the latter from the fields of manufacturing and astronomy, obtaining encouraging results.