Automated Data Cleaning via Multi-View Anomaly Detection

One of the first steps in the data analysis pipeline is data cleaning: detecting data from failed sensors. This talk will discuss the application of anomaly detection algorithms to find and remove bad readings from weather station data. We will review our previous work on DBN time series models and our current work on applying non-parametric anomaly detection algorithms as part of our SENSOR-DX multi-view anomaly detection architecture. A major challenge in evaluating these algorithms is to obtain ground truth, because real sensor data tends to be labeled conservatively by domain experts.