Skip to content

[AINode]: Integrate MOMENT as a builtin forecasting model #17386

Open
Eliaaazzz wants to merge 3 commits intoapache:masterfrom
Eliaaazzz:f_ainode_moment
Open

[AINode]: Integrate MOMENT as a builtin forecasting model #17386
Eliaaazzz wants to merge 3 commits intoapache:masterfrom
Eliaaazzz:f_ainode_moment

Conversation

@Eliaaazzz
Copy link
Copy Markdown

Description

Integrate MOMENT Time Series Foundation Model

Integrate MOMENT (A Family of Open Time-series Foundation Models, ICML 2024, Auton Lab/CMU) into IoTDB AINode as a built-in forecasting model.

MOMENT uses a T5 encoder-only backbone with patch-based input embedding and RevIN (Reversible Instance Normalization). It supports zero-shot forecasting with a fixed input length of 512 timesteps
and channel-independent multivariate processing.

Architecture:
Input [batch, n_channels, 512]
→ RevIN normalization
→ Patching (patch_len=8, stride=8 → 64 patches)
→ Patch embedding (linear → d_model)
→ T5 Encoder (self-attention)
→ ForecastingHead (linear → forecast_horizon)
→ RevIN denormalization
→ Output [batch, n_channels, forecast_horizon]

The default variant is MOMENT-1-small (~152 MB, d_model=512, 6 layers). Weights are downloaded from HuggingFace Hub on first use, consistent with existing built-in models (Sundial, Timer-XL,
Chronos2, etc.).

Design Decisions

  • Vendor the model code rather than depending on the external momentfm pip package, following the same pattern as Chronos2 ([AINode] Integrate Chronos2 as builtin forecasting model #16903) and Toto ([AINode]: Integrate toto as a builtin forecasting model #17322). This avoids an additional runtime
    dependency and gives full control over the loading process.
  • Wrap as PreTrainedModel with custom from_pretrained() to handle upstream MOMENT's flat state-dict key mapping (prefixing with moment.*) and T5 config extraction from config.json.
  • Use MOMENT-1-small as the default built-in variant to minimize download size and memory usage while still providing useful forecasting capability.
  • Iterative long-horizon forecasting in the pipeline: for output_length > model_forecast_horizon, predictions are appended to the context window and fed back as input, similar to
    autoregressive decoding.
  • Fixed 512-length input handling: inputs shorter than 512 are left-padded with zeros and masked; inputs longer than 512 are truncated to the last 512 timesteps.

Bug Fix

Fixed unquoted $JAVA variable in confignode-env.sh and datanode-env.sh to handle paths with spaces (e.g. C:\Program Files\...).


This PR has:

  • been self-reviewed.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • iotdb-core/ainode/iotdb/ainode/core/model/moment/ — New MOMENT model package
    • configuration_moment.pyMomentConfig(PretrainedConfig)
    • modeling_moment.pyMomentForPrediction(PreTrainedModel) with RevIN, Patching, T5 Encoder, ForecastingHead
    • pipeline_moment.pyMomentPipeline(ForecastPipeline) with padding/truncation and iterative forecasting
  • iotdb-core/ainode/iotdb/ainode/core/model/model_info.py — Added moment to BUILTIN_HF_TRANSFORMERS_MODEL_MAP
  • iotdb-core/ainode/iotdb/ainode/core/constant.py — Added memory estimate for MOMENT
  • integration-test/.../AINodeTestUtils.java — Added moment to BUILTIN_LTSM_MAP
  • LICENSE — Added MIT attribution for MOMENT source code
  • scripts/conf/confignode-env.sh, scripts/conf/datanode-env.sh — Fixed unquoted $JAVA variable

Copy link
Copy Markdown
Contributor

@CRZbulabula CRZbulabula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Elia, this is your first PR (pull request) for Apache IoTDB repository, our community highly appreciate your contribution!

Next, let us talk about this PR, the following points still need to be improved:

  1. The moment model has serveral versions, we can employ the large version.
  2. The format of AINode should be fixed. Run black . && isort . under iotdb-core/ainode.
  3. The forecast IT failed, you should ensure the correctness first in your local environment.
  4. The modification in constant.py should be reverted, since we do not support the corresponding function in current version.

- Switch from MOMENT-1-small to MOMENT-1-large (repo_id, d_model, backbone)
- Fix from_pretrained to handle loader kwargs (config, trust_remote_code)
- Store t5_config in MomentConfig for proper d_model inference
- Add shape check when filtering state dict to skip mismatched head weights
- Revert constant.py modification (unsupported in current version)
- Apply black and isort formatting
@Eliaaazzz
Copy link
Copy Markdown
Author

Hi Elia, this is your first PR (pull request) for Apache IoTDB repository, our community highly appreciate your contribution!

Next, let us talk about this PR, the following points still need to be improved:

  1. The moment model has serveral versions, we can employ the large version.
  2. The format of AINode should be fixed. Run black . && isort . under iotdb-core/ainode.
  3. The forecast IT failed, you should ensure the correctness first in your local environment.
  4. The modification in constant.py should be reverted, since we do not support the corresponding function in current version.

Thanks for the review and warm welcome. I will fix these in the next update.

  • I will use the larger model version when multiple versions are available.
  • I will run black . && isort . under iotdb-core/ainode and fix the AINode formatting issues.
  • I will reproduce the forecast integration test failure locally and make sure it passes before pushing the next revision.
  • I will revert the change in constant.py, since that function is not supported in the current version.

@Eliaaazzz
Copy link
Copy Markdown
Author

I‘m still trying to reproduce the forecast IT errors and I'll push a new commit right after I fixed the errors. Please bear with me.

RevIN was computing mean/stdev over the full 512-length tensor including
zero-padded positions, corrupting normalization statistics for inputs
shorter than 512 timesteps. Pass the input_mask through to RevIN so
statistics are computed only over observed positions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants