Skip to content

v4.5.0 ClickHouse writers require CH ≥25.8, but the Helm chart still bundles ClickHouse 25.7.5 → every insert fails with UNKNOWN_SETTING #4197

Description

@brentshulman-silkline

Summary

v4.5.0's ClickHouse writers (internal-packages/clickhouse/src/{taskRuns,taskEvents,metrics,sessions}.ts) unconditionally set input_format_json_infer_array_of_dynamic_from_array_of_different_types: 1 on every insert. That setting was introduced in ClickHouse 25.8. The Trigger Helm chart (oci://ghcr.io/triggerdotdev/charts/trigger) still bundles ClickHouse via the Bitnami subchart at 25.7.5 (older chart versions: 25.6.1). On any server < 25.8, every insert is rejected:

Code: 115 … input_format_json_infer_array_of_dynamic_from_array_of_different_types
is neither a builtin setting nor started with the prefix 'SQL_' (UNKNOWN_SETTING)

Result: run execution is unaffected (Postgres is the execution source of truth), but every ClickHouse-backed read path — the dashboard run list, metrics, and the list-runs / run-status APIs — silently freezes, and failed insert batches are dropped after retries (permanent run-history gaps). The setting is clearly intentional (there's a regression test in taskRuns.test.ts guarding mixed-type JSON arrays from nested-Tuple inference / CH Code 117 during merges), so this is a version-floor / packaging issue, not a request to remove the setting.

Environment

  • Self-hosted v4 on Kubernetes (EKS)
  • Upgraded chart + webapp/supervisor images 4.4.6 → 4.5.0
  • ClickHouse 25.7.5 (the chart's bundled Bitnami subchart)
  • Had to roll back to 4.4.6 to recover. v4.5.1 does not fix it (its only change is WorkOS Directory Sync / SCIM).

Requests

  1. Bump the chart's bundled ClickHouse subchart to ≥25.8, or gate the setting on the server version so it degrades gracefully on older servers.
  2. Document the minimum ClickHouse version for v4.5.x in the self-hosting docs. This is the second time the bundled-CH version has bitten self-hosters — see Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart #3520, where the 25.6.1 memory-tracker overflow led to the bump to 25.7.5.
  3. Guidance: now that Bitnami's free image catalog is EOL (bitnamilegacy is a frozen archive with no ≥25.8 tag), what is the supported ClickHouse image path for the chart going forward — a Bitnami Secure Images tag, the official clickhouse/clickhouse-server image, or something else?

Happy to test a chart bump and report back, as with #3520.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions