4.1.3 Release Note

[yiguolei]

4.1.2 release notes

New Features

Query & Execution

• Support Python UDF/UDAF/UDTF (branch-4.1: [Feat](udf) Support Python UDF/UDAF/UDTF #63387)
• Support remote scan bytes breaker in workload policy ([feature](workload) Support remote scan bytes breaker in workload policy #64649)
• Add username-based backend workload policy support ([enhancement](workload policy) Add username-based backend workload policy support #60559)
• Add is_valid_utf8 function ([feature](function) support is_valid_utf8 function #62515)

Cloud Native

• Add table-level event-driven warm up ([feature](cloud) Add table-level event-driven warm up #63832)

Load

• Support zstd stream load compression ([feature](be) Support zstd stream load compression #64711)

Index & Search

• Add ANN TopN small-candidate fallback to brute-force search ([feature](ann-index) Add ann topn small candidate fallback session va… #64555)

Improvement

Query & Execution

• Treat inactive_file as available memory to avoid false query cancellations in cgroup environments ([improvement](cgroup) inactive_file should be treated as available memory to avoid query be cancelled #64347)
• Set default profile level to 2 to print more detail query profile by default([improvement](fe) Set default profile level to 2 #64378)
• Add configurable return mode for insert publish timeout in ETL scenarios (branch-4.1: [improvement](fe) Add configurable return mode for insert publish timeout in ETL scenarios #64583)

Load

• Make from-to streaming task timeout progress-aware ([improve](streaming-job) make from-to streaming task timeout progress-aware #64301)
• Add zero-row hint for Kafka read_committed load and delay zero-row retries (branch-4.1: [enhance](job) add zero-row hint for Kafka read_committed load and [opt](job) delay Kafka read committed zero-row retries #64585)
• Refresh routine load lag metrics more timely ([enhance](job) refresh routine load lag more timely #63654)

Cloud Native

• Add rate limit for BE to Meta-Service RPC (branch-4.1: [Opt](cloud) Add rate limit for BE to MS rpc #64396)
• Add virtual compute group switch metric ([improvement](fe) Add virtual compute group switch metric #63036)

Bugfix

Index & Search

• Handle truncated sparse path stats when reading Variant ([fix](variant) Handle truncated sparse path stats when reading variant #64205)
• Preserve JSON object when casting JSONB to Variant ([fix](variant) Preserve json object when casting jsonb to variant #63792)
• Return raw string for element_at on scalar-string Variant ([fix](variant) return raw string for element_at on scalar-string variant #64103)
• Escape JSONB path member control characters ([fix](be) Escape JSONB path member control characters #63517)

Query & Execution

• Fix SimplifyAggGroupBy should verify injectivity ([fix](simplify agg) SimplifyAggGroupBy should verify injectivity #64335)
• Fix partition TopN optimization requiring window function partition key to be subset of co-located keys (branch-4.1: [fix](nereids) partition topn opt requires chosen window func partition key to be a subset of co-located ones #65073)
• Fix sorted merge continuing when sender queue is ready ([fix](be) Continue sorted merge when sender queue is ready #65004)
• Fix TopN merge with child prefix order keys ([fix](fe) Merge TopN with child prefix order keys #64685)
• Fix nullable aggregate visitor dispatch ([fix](aggregate) Fix nullable aggregate visitor dispatch #64885)
• Fix incorrect aggregate merge with duplicate aliases ([fix](agg) Fix incorrect aggregate merge with duplicate aliases #65025)
• Fix timeout checker stopping after an exception ([fix](connection) Prevent timeout checker from stopping after an exception #65040)
• Restrict auto salt join rewrite to avoid incorrect results ([fix](join) Restrict auto salt join rewrite #64518)
• Fix handling of generated columns in delete partial update ([fix](fe) Handle generated columns in delete partial update #64884)
• Guard Count(*) child access with arity check in PushDownAggThroughJoinOnPkFk ([fix](agg push down) guard Count(*) child access with arity() check in PushDownAggThroughJoinOnPkFk #64848)
• Fix cloud query retry setting version_cache_ttl temporarily to 0 on error -230 ([fix](fe) set cloud version_cache_ttl to 0 temporarily if retry a query with -230 #63721)
• Fix recursive CTE blocks not sent to every scan instance ([fix](fe) Send recursive CTE blocks to every scan instance #64964)
• Fix SHOW PROCESSLIST returning unexpected result in FULL mode ([fix](process list) return unexpected result in FULL mode #64631)
• Fix isNull parser placement under primaryExpression ([fix](parser) isNull should under primaryExpression #63619)
• Fix subquery eliminating null-aware anti-join incorrectly ([fix](subquery) eliminate null aware by mistake #64639)
• Skip TopN lazy materialization when light_schema_change=false ([fix](topn) Skip TopN lazy materialization when light_schema_change=false #64441)
• Fix copy into select not binding file column placeholders (branch-4.1: [fix](nereids) bind file column placeholders for copy into select #64591)
• Clamp merged limit of MERGE_TOP_N by parent offset ([fix](nereids) clamp the merged limit of MERGE_TOP_N by the parent offset #64306)
• Prevent cast project pushdown through UNION DISTINCT ([fix](fe) Prevent cast project pushdown through union distinct #64080)
• Disable row-store lazy fetch for shared base columns ([fix](fe) Disable row-store lazy fetch for shared base columns #62864)
• Fix self-deadlock in FragmentMgr on query context map clear (branch-4.1: [fix](fragment) avoid query-ctx map clear self-deadlock when stop FragmentMgr #64552)
• Fix month nullable datetimev2 literal binding ([Fix](nereids) Fix month nullable datetimev2 literal binding #64459)
• Remove incorrect topN-to-MAX optimizer rewrite ([fix](topn_to_max)Remove the topnToMax optimizer rewrite that converted TOPN into MAX. #63519)
• Fix local runtime filter merge deadlock ([fix](be) Avoid local runtime filter merge deadlock #65102)
• Return error status from materialization operator row-ID fetcher ([Exec](status) materialization_opertor return the error status by row_id_fetcher #62513)
• Normalize aggregate order-by pushdown ([fix](analyzer) Normalize aggregate order by pushdown #64787)
• Fix NereidsCoordinator not created in proxy flow due to missing parsedStatement ([fix](fe) Fix NereidsCoordinator not created in proxy flow due to missing parsedStatement on StatementContext #64363)
• Reject more than 1 argument in COUNT DISTINCT window function ([fix](window) only allow 1 argument in count distinct in window function #64783)
• Avoid incorrect convert_tz partition pruning across DST transitions ([fix](partition_prune) Avoid incorrect convert_tz partition pruning across DST transitions #63853)
• Fix MAKE_SET constant folding not clearing prior result ([Fix](fold_const) MAKE_SET constant folding should clear #64907)
• Prevent SIGFPE in int_divide for signed minimum value ([fix](be) prevent SIGFPE in int_divide for signed min value #64828)
• Fix nth_value for upper-bounded windows ([fix](be) Fix nth_value for upper bounded windows #64864)
• Return NaN for avg_weighted when sum of weights is zero ([fix](be) Return NaN for avg_weighted when sum of weights is zero #64333)
• Stabilize conjunct cost ordering ([fix](be) Stabilize conjunct cost ordering #64637)
• Stop extra operator work after cancellation ([fix](be) Stop extra operator work after cancellation #64077)
• Fix array_first/array_last boolean cast ([fix](array function) Support boolean cast for array first and last #64847)
• Validate array_sort lambda arity ([fix](function) Validate array_sort lambda arity #64825)
• Limit retention param count to 32 to avoid BE heap overflow ([fix](retention) Limit param count to 32 to avoid BE heap overflow #64521)
• Fix concat_ws nullable array handling ([fix](be) Fix concat_ws nullable array handling #64703)
• Fix split_by_delimiter missing backslash escape handling ([fix](serde) fix split_by_delimiter missing backslash escape handling #61995)
• Normalize v1 date string cast result ([fix](be) Normalize v1 date string cast result #64575)
• Fix SIGSEGV in bvar::take_sample under high EPS due to AgentCombiner/TLS race ([fix](be) Fix SIGSEGV in bvar::take_sample caused by AgentCombiner/TLS Agent lifetime race under high EPS #64040)
• Validate sequence pattern event numbers ([fix](be) Validate sequence pattern event numbers #64930)
• Catch standard exceptions in pipeline scheduler ([fix](be) Catch std exceptions in pipeline scheduler #65019)
• Catch block serialization exceptions to avoid coredump ([fix](be) catch block serialization exceptions to avoid coredump #64852)
• Validate task executor scan handles ([fix](be) Validate task executor scan handles #65054)
• Initialize thread context on AsyncIO worker threads ([fix](be) Init thread context on AsyncIO worker threads #64846)
• Fix sliced FixedSizeBinary Arrow string reads ([fix](be) Fix sliced FixedSizeBinary Arrow string reads #64829)
• Avoid mutable nullable CRC32C hashing ( [fix](column) avoid mutable nullable crc32c hashing #64944)
• Preserve DateTimeV2 scale in OUTFILE CSV export ([fix](outfile)Preserve DateTimeV2 scale in OUTFILE CSV. #64344)
• Reject invalid IPv4 default value at CREATE TABLE time ([fix](ddl) reject invalid IPv4 default value at CREATE TABLE time #62906)
• Harden Arrow Flight split source error path to avoid BE crash on external scan ([fix](arrow-flight) Harden split source error path to avoid BE crash on external table scan #64797)

Storage & Compaction

• Fix group commit block queue unavailability ([fix](group commit) fix can not get a block queue #63722)
• Fix lost rows when prepared statement reuses plan with same load_id in group commit ([fix](group commit) fix lost row when prepared stmt reused plan shares one load_id #64362)
• Fix shared delta writer state lifetime (branch-4.1: [fix](delta writer) Fix shared delta writer state lifetime #64504)
• Fix tablet writer map lookup race in load channel ([fix](be) Protect tablet writer map lookup in load channel #64604)
• Prevent EMPTY_CUMULATIVE / BASE-CUMU races on same tablet in cloud compaction ([fix](cloud-compaction) prevent EMPTY_CUMULATIVE / BASE-CUMU races on the same tablet #64619)
• Persist update time for sub-transaction commit ([fix](cloud) Persist update time for sub txn commit #64739)
• Make S3 rate limiter config take effect dynamically in cloud mode ([fix](cloud) make S3 rate limiter config take effect dynamically in cloud mode #64554)
• Use snapshot read for table version to avoid transaction conflict ([fix](cloud) use snapshot read for table version to avoid txn conflict #64647)
• Recycle empty rowsets without resource ID ([fix](cloud) Recycle empty rowsets without resource id #64630)
• Fix double assignment in recycler_service that caused recycler malfunction ([fix](cloud) Fix double assignment in recycler_service.cpp that cause… #64168)
• Avoid false-positive leaked delete bitmaps for unexpired job tmp rowsets ([fix](checker) Avoid false-positive leaked delete bitmaps for unexpired job tmp rowsets #64313)
• Refresh base tablet before schema change V1 ([fix](cloud) Refresh base tablet before schema change V1 #64312)
• Forbid restoring table with light_schema_change=false in cloud mode ([bug](cloud restore) forbid restoring table with property 'light_schema change' = 'false' in cloud mode #62914)
• Rewrite table properties and partition info during cloud restore (branch-4.1: [bug](cloud restore) rewrite table properties and partition info in cloud restore #64466)
• Make sync file cache clear use safe removal ([fix](file-cache) Make sync clear use safe removal #64578)
• Disable sync file cache clear in HTTP API ([fix](filecache) Disable sync file cache clear in http api #64321)
• Exclude warmup reads from file cache hit ratio metrics ([fix](filecache) exclude warmup reads from file cache hit ratio metrics #63394)
• Force drop partition in INSERT OVERWRITE ([fix](iow) force drop partition in INSERT OVERWRITE #62510)
• Add null-safety to getBaseViewsOneLevel for MTMV backward compatibility ([fix](mtmv) Add null-safety to getBaseViewsOneLevel for backward compatibility #64412)
• Fix MTMV mutating excluded trigger tables ([fix](mtmv) Avoid mutating excluded trigger tables #62984)
• Avoid finalized pipeline task submit crash ([fix](be) Avoid finalized pipeline task submit crash #64953)

Load

• Serialize audit loader batch assembly to fix race ([fix](audit) serialize audit loader batch assembly #65107)
• Lock routine load task renew on submit failure (branch-4.1: [fix](job) lock routine load task renew on submit failure #65007)
• Fix stream load IPv6 host parsing ([fix](stream_load) Fix stream load IPv6 host parsing #64147)
• Replace tablet writer close polling with event wakeup ([fix](load) replace tablet writer close polling with event wakeup #64221)
• Fix empty statistics for forwarded INSERT ([fix](load) fix empty statistics for forwarded INSERT #64439)
• Fix load_to_single_tablet routing for auto partition ([fix](load) fix load_to_single_tablet routing for auto partition #64356)
• Record per-query SET_VAR hint session variables in audit log ([fix](audit) record per-query SET_VAR hint session variables in audit log #64569)
• Fix AVRO JNI reader coredump with null pointer during TVF function ([bug](avro) fix avro jni reader coredump with nullptr #64699)

Lakehouse

• Fix NPE in Iceberg COUNT() pushdown when snapshot summary omits total- counters ([fix](iceberg) Fix NPE in COUNT(*) pushdown when snapshot summary omits total-* counters #64648)
• Fix Iceberg binary columns written with wrong Arrow types ([fix](iceberg) Write binary columns with proper Arrow types #64949)
• Add LZ4 compression support for Iceberg/Hive Parquet/ORC writers ([fix](iceberg)Support LZ4 compression for iceberg/hive Parquet/ORC writers. #64723)
• Allow disabling REST catalog view operations (branch-4.1: [fix](iceberg) Allow disabling REST catalog view operations #63320)
• Preserve empty text records when reading Hive text files ([fix](hive) Preserve empty text records #64671)
• Estimate MaxCompute write block size from Arrow buffers instead of per-row serialization ([fix](maxcompute) Estimate write block size from Arrow buffers, not per-row serialization #64612)
• Push SQL Server/Oracle boolean predicates as 1/0 instead of TRUE/FALSE in JDBC (branch-4.1: [fix](jdbc) Push SQL Server/Oracle boolean predicates as 1/0 instead of TRUE/FALSE #64757)
• Fetch table comment PROP from Iceberg table metadata ([iceberg](fix) Fetch comment PROP from table meta #64263)
• Avoid blocking external meta cache refresh on slow miss load ([fix](fe) Avoid blocking external meta cache refresh on slow miss load #64705)

Security & Authentication

• Add auth check for manager node and query error REST APIs (branch-4.1: [fix](auth) add auth check for manager node and query qerror REST APIs #65080)