Distributed Event Processing and Load Balancing

Overview

This repository implements a distributed event aggregation framework with load balancing optimization for Bachelor's thesis work. The system provides plan generation, cost modeling, and execution capabilities for complex event processing (CEP) queries across distributed networks, including Raspberry Pi clusters.

Key Features

• DIPSUM Plan Generation: Distributed event aggregation plan generation with multiple optimization strategies
• Load Balancing Optimization: Novel cost models that balance network traffic and computational load across distributed nodes
• Simulation Framework: Local and distributed simulation capabilities for performance analysis
• Pi Cluster Deployment: Real-world experiments on Raspberry Pi cluster infrastructure
• Cost Analysis: Comprehensive cost models for message passing and aggregate computation

Project Structure

Core Modules

• src/: Core source code modules

  • DIPSUM/: DIPSUM plan generator with cost models and sensitivity experiments

    • C++ plan enumerator for exhaustive plan enumeration
    • Multiple plan generation strategies (C&C, Min-Algorithm, BF DIPSUM)
    • Cost models and sensitivity analysis

  • LoadBalancing/: Load balancing optimization model

    • Balanced plan generation considering both network and compute costs
    • Sensitivity analysis and plotting utilities

  • utils/: Analysis and visualization utilities

    • Experiment data evaluation and statistical analysis
    • Plotting tools for timing results and resource limits

• experiments/: Simulation and experimental infrastructure

  • simulation/: Local simulation framework with event trace generation
  • Pi-Cluster-Experiment/: Real-world experiments on Raspberry Pi cluster
  • local_simulation.py: Local simulation with execution time measurements
  • pi_cluster_simulation.py: Pi cluster orchestration and deployment

Main Entry Points

• main.py: Primary orchestration for experiments and analysis
• experiments/local_simulation.py: Local simulation with execution time measurements
• experiments/pi_cluster_simulation.py: Raspberry Pi cluster orchestration and deployment

Dependencies

pip install -r requirements.txt

Core dependencies: paramiko, numpy, matplotlib, scipy

System packages (optional): sshpass, jq

Usage

Running Experiments

python main.py

Local Simulation

python experiments/local_simulation.py

Pi Cluster Experiments

python experiments/pi_cluster_simulation.py

Ensure SSH access and proper network configuration for Pi cluster experiments.

Sensitivity Analysis

Navigate to DIPSUM/sensitivity_experiment.py or LoadBalancing/sensitivity.py to configure and run sensitivity experiments.

Datasets

The repository includes experiment configurations for:

• BostonCrime: Crime incident data
• NASDAQ: Stock market trade data
• Synthetic traces: Generated event streams with configurable parameters

Research Context

This work focuses on optimizing distributed event aggregation by:

1. Minimizing network communication overhead
2. Balancing computational load across nodes
3. Supporting multi-sink query patterns
4. Providing practical deployment on resource-constrained devices