Tools

• Data pipelines

Data pipelines

The Tools module includes a mini-framework for building development-friendly data pipelines. By "development-friendly" we mean that you can build and run pipelines with a minimum of fuss.

Let's look at an example.

The pipeline

This sample pipeline compares two anomaly detection models: EWMA and PEWMA. Here's how it works:

• The random walk metric source generates random walk time series data.
• The EWMA anomaly detector applies the EWMA model to the time series to find anomalies.
• The EWMA RMSE evaluator measures the performance of the EWMA anomaly detector using root mean square error.
• The EWMA chart presents the metric data, the anomaly detection model and classifications, and the RMSE.
• Similarly for the PEWMA anomaly detector, evaluator and chart.

The code

The following code shows how to wire all of this up:

public class RandomWalkEwmaVsPewma {
    
    public static void main(String[] args) {
        final RandomWalkMetricSource source = new RandomWalkMetricSource();
        
        final AnomalyDetectorFilter ewmaAD =
            new AnomalyDetectorFilter(new EwmaAnomalyDetector());
        final AnomalyDetectorFilter pewmaAD =
            new AnomalyDetectorFilter(new PewmaAnomalyDetector());
        
        final EvaluatorFilter ewmaEval =
            new EvaluatorFilter(new RmseEvaluator());
        final EvaluatorFilter pewmaEval =
            new EvaluatorFilter(new RmseEvaluator());
    
        final AnomalyChartSink ewmaChart =
            PipelineFactory.createChartSink("EWMA");
        final AnomalyChartSink pewmaChart =
            PipelineFactory.createChartSink("PEWMA");
    
        source.addSubscriber(ewmaAD);
        ewmaAD.addSubscriber(ewmaEval);
        ewmaAD.addSubscriber(ewmaChart);
        ewmaEval.addSubscriber(ewmaChart);
    
        source.addSubscriber(pewmaAD);
        pewmaAD.addSubscriber(pewmaEval);
        pewmaAD.addSubscriber(pewmaChart);
        pewmaEval.addSubscriber(pewmaChart);
        
        showChartFrame(createChartFrame(
            "Random Walk",
            ewmaChart.getChart(),
            pewmaChart.getChart()));
        source.start();
    }
}

The result

When you run the pipeline, you should see something like the following: