Detecting anomalies in your data is a well-studied challenge. One way of tackling it is through machine learning (ML). By training a model on ground truth examples of normal and abnormal data, it can be taught to distinguish the former from the latter.
However, working with industrial customers we’ve noticed that normal data examples vastly outnumber abnormal ones. In extreme cases anomalous data is not present at all as new machines are put into production and failures have yet to emerge.
So, should we rule out ML as a tool in such scenarios? Not necessarily, what if we change the ML task definition? We come from a binary classification task, “is this normal, yes, or no?” Let’s change that to: “how normal is this data?” This subtle change allows us to take a completely different approach. We can train a model on nothing but normal healthy data and evaluate it on how well it can reconstruct said data. When the data is behaving in a normal, predictable, way, we can expect this model to have rather accurate predictions, but when the data starts to deviate from the normal data the model was trained on its predictions will become less accurate. The accuracy of our model thus becomes a proxy for how normal the input data is which in combination with a threshold can be used as a trigger for anomalies.
