DTLMod/ChangeLog at main · simgrid/DTLMod · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
----------------------------------------------------------------------------

DTLMod (0.5) not released yet (target: May 2026)

Improvements:
  - Memory efficiency: progressive eviction of metadata transaction entries
    - Once all subscribers have consumed a transaction, its entries are evicted
      from the in-memory Metadata::transaction_infos_ map
    - When metadata export is enabled and publishers and subscribers coexist
      (file streaming), evicted entries are progressively flushed to
      per-variable temporary files; the final metadata file is assembled at
      pub_close() from those files and any remaining in-memory entries,
      preserving the existing format and transaction count
    - When the stream is opened by subscribers only after all publishers have
      closed (sequential scenario), memory-only eviction is performed (the
      metadata file has already been written by pub_close())
    - Memory footprint of the File engine now grows as O(N_pub) instead of
      O(N_pub × N_transactions) for long-running concurrent streaming workloads

----------------------------------------------------------------------------

DTLMod (0.4) February 16, 2026

Major improvements:
  - Data Reduction framework
    - New abstract ReductionMethod interface for extensible data reduction strategies
    - Decimation method: spatial subsampling with per-dimension stride, optional
      interpolation (linear, quadratic, cubic), configurable computational cost,
      and support for both publisher-side and subscriber-side application
    - Compression method: size reduction with preserved shape, supporting three
      compressor profiles (fixed ratio, SZ-inspired, and ZFP-inspired models),
      separate compression and decompression costs, and per-transaction ratio
      variability
    - Publisher-side reduction state is propagated to subscribers through
      inquire_variable, enabling detection and prevention of conflicting
      double reductions
    - Reduction operations work with both File and Staging engines
  - New documentation pages for the reduction feature (Reduction, Decimation,
    and Compression) and updated Main Concepts page
  - Improved test coverage
    - Comprehensive C++ and Python tests for both reduction methods
    - Coverage of error handling, parameter validation, re-parameterization,
      and publisher-subscriber workflows for each reduction method
    - New test for subscriber-first arrival pattern in File engine
  - Code quality and CI improvements
    - Improved coverage reporting with SonarQube and CodeFactor integration
    - Added subscriber synchronization barrier in File engine
    - Removed unnecessary defensive guards and race condition checks that
      cannot occur under SimGrid's maestro orchestration

API Changes:
  - New Stream method:
    - Stream::define_reduction_method(name) creates a named reduction method
      ("decimation" or "compression") for the stream
  - New Variable methods and properties:
    - Variable::set_reduction_operation(method, parameters) applies a reduction
      with key-value parameters to a variable
    - Variable::is_reduced(), is_reduced_by_publisher(), is_reduced_by_subscriber()
      query the reduction state
    - Variable::get_reduction_method() retrieves the applied ReductionMethod
  - New ReductionMethod class with query methods:
    - get_reduced_variable_global_size(), get_reduced_variable_local_size()
    - get_reduced_variable_shape()
    - get_flop_amount_to_reduce_variable(), get_flop_amount_to_decompress_variable()
  - Full Python bindings for all reduction operations, including nine new
    exception types for parameter validation errors
  - Engine::put() now automatically simulates reduction cost and transports the
    reduced data size when the variable is reduced
  - Engine::get() now automatically simulates decompression cost after receiving
    compressed data

----------------------------------------------------------------------------

DTLMod (0.3) January 19, 2026

Major improvements:
  - Enhanced code quality with comprehensive refactoring
    - Improved modern C++17 usage throughout the codebase
    - Reduced code complexity and technical debt
  - Memory safety and robustness improvements
    - Fixed critical memory safety issue in DTL connection management
    - Fixed race condition in Engine creation
    - Plugged memory leaks
  - Major refactoring with improved memory management, encapsulation, and move-only semantics
  - Improved CI/CD infrastructure
    - Added Valgrind and sanitizer checks (AddressSanitizer, UndefinedBehaviorSanitizer)
    - Enhanced test coverage and validation
  - Comprehensive proofread of the documentation

API Changes:
  - DTL::get_stream_by_name_or_null() renamed to DTL::get_stream_by_name()
    Returns std::optional<std::shared_ptr<Stream>> instead of raw pointer
    Python: dtl.get_stream_by_name() now returns None if stream not found
  - Metadata management transferred from Engine to Stream
    Stream::get_metadata_file_name() replaces Engine::get_metadata_file_name()
    Python: stream.metadata_file_name replaces engine.metadata_file_name
  - Stream method chaining improvements
    set_engine_type() and set_transport_method() now return Stream& instead of Stream*
    Enables cleaner fluent interface: stream.set_engine_type(...).set_transport_method(...)
  - DTL::create() now accepts std::string_view and has empty default parameter
  - String parameters changed to std::string_view for better performance:
    - DTL::add_stream(), DTL::get_stream_by_name()
    - Stream and Variable constructors and methods
  - get_engine_type_str() and get_transport_method_str() now return std::optional<const char*>

----------------------------------------------------------------------------

DTLMod (0.2) November 11, 2025

  - New comprehensive documentation
  - Full python bindings
  - Improved code coverage
  - Reduced technical debt
  - Fixed deadlocks in File engine
  - Update to SimGrid 4.1 and FSMod 0.4
  - API:
    - Make metadata export configurable at stream level.
    - Have a simpler version of Engine::put() that automatically uses the
      local size of the Variable.
    - Stream::remove_variable now throws an UnknownVariableException rather
      than returning a bool

----------------------------------------------------------------------------

DTLMod (0.1) July 4, 2025

First Release

----------------------------------------------------------------------------