multiline: add built-in json parser for JSON multiline objects by lecaros · Pull Request #12264 · fluent/fluent-bit

lecaros · 2026-08-11T19:55:10Z

Add built-in multiline parser json for tail inputs that read JSON objects split across lines.

The parser groups lines only (regex start ^\{.*, continuation on
whitespace-indented lines or a lone }). It does not embed a JSON parser;
per-line JSON parsing fails on partial lines such as { or "id": 102,.

Expected usage:

Pretty-printed / mixed files: multiline.parser: json on tail, then
filter_parser with key_name: log and parser: json.
JSON Lines (one object per line): parser: json on tail; no multiline
parser.

Addresses #2418, #8232

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

config

service:
  flush: 1
  log_level: debug
parsers:
  - name: json
    format: json
    time_key: time
    time_format: "%Y-%m-%dT%H:%M:%S.%L"

pipeline:
  inputs:
    - name: tail
      tag: json.oneline
      path: ./oneline.json
      parser: json
      read_from_head: true
      path_key: file

    - name: tail
      tag: json.mixed
      path: ./mixed.json
      read_from_head: true
      multiline.parser: json
      path_key: file

  filters:
    - name: parser
      match: json.mixed
      key_name: log
      parser: json
      preserve_key: false

  outputs:
    - name: stdout
      match: json.*
      format: json_lines

oneline.json

{"id":1,"time":"2026-08-11T10:00:00.000","level":"info","msg":"oneline record one"}
{"id":2,"time":"2026-08-11T10:00:01.000","level":"warn","msg":"oneline record two"}
{"id":3,"time":"2026-08-11T10:00:02.000","level":"error","msg":"oneline record three"}

mixed.json

{"id":101,"time":"2026-08-11T11:00:00.000","level":"info","msg":"single-line record A"}
{
  "id": 102,
  "time": "2026-08-11T11:00:01.000",
  "level": "warn",
  "msg": "multiline record B",
  "details": {
    "user": "alice",
    "action": "login"
  }
}
{"id":103,"time":"2026-08-11T11:00:02.000","level":"info","msg":"single-line record C"}
{"id": 104,
  "time": "2026-08-11T11:00:03.000",
  "level": "error",
  "msg": "multiline record D",
  "stack": [
    "frame1",
    "frame2"
  ]
}
{"id":105,"time":"2026-08-11T11:00:04.000","level":"info","msg":"single-line record E"}

- [x] Debug log output from testing the change

debug output

Fluent Bit v5.1.0
* Copyright (C) 2015-2026 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |  ___||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   _|___ \ | |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \|  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V //\__/ /\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)\___/


[2026/08/11 14:34:16.730] [ info] Configuration:
[2026/08/11 14:34:16.730] [ info]  flush time     | 1.000000 seconds
[2026/08/11 14:34:16.730] [ info]  grace          | 5 seconds
[2026/08/11 14:34:16.730] [ info]  daemon         | 0
[2026/08/11 14:34:16.730] [ info] ___________
[2026/08/11 14:34:16.730] [ info]  inputs:
[2026/08/11 14:34:16.730] [ info]      tail
[2026/08/11 14:34:16.730] [ info]      tail
[2026/08/11 14:34:16.730] [ info] ___________
[2026/08/11 14:34:16.730] [ info]  filters:
[2026/08/11 14:34:16.730] [ info]      parser.0
[2026/08/11 14:34:16.730] [ info] ___________
[2026/08/11 14:34:16.730] [ info]  outputs:
[2026/08/11 14:34:16.730] [ info]      stdout.0
[2026/08/11 14:34:16.730] [ info] ___________
[2026/08/11 14:34:16.730] [ info]  collectors:
[2026/08/11 14:34:16.730] [ info] [fluent bit] version=5.1.0, commit=ae51533a2d, pid=70087
[2026/08/11 14:34:16.731] [debug] [engine] coroutine stack size: 65536 bytes (64.0K)
[2026/08/11 14:34:16.731] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2026/08/11 14:34:16.731] [ info] [simd    ] NEON
[2026/08/11 14:34:16.731] [ info] [cmetrics] version=2.2.1
[2026/08/11 14:34:16.731] [ info] [ctraces ] version=0.7.1
[2026/08/11 14:34:16.731] [ info] [input:tail:tail.0] initializing
[2026/08/11 14:34:16.731] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2026/08/11 14:34:16.731] [debug] [tail:tail.0] created event channels: read=21 write=22
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] scanning path ./oneline.json
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] inode=1288433329 with offset=0 appended as ./oneline.json
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] scan_glob add(): ./oneline.json, inode 1288433329
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] 1 new files found on path './oneline.json'
[2026/08/11 14:34:16.731] [ info] [input:tail:tail.1] initializing
[2026/08/11 14:34:16.731] [ info] [input:tail:tail.1] storage_strategy='memory' (memory only)
[2026/08/11 14:34:16.731] [debug] [tail:tail.1] created event channels: read=28 write=29
[2026/08/11 14:34:16.731] [ info] [input:tail:tail.1] multiline core started
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] flb_tail_fs_stat_init() initializing stat tail input
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] scanning path ./mixed.json
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] inode=1288433424 with offset=0 appended as ./mixed.json
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] scan_glob add(): ./mixed.json, inode 1288433424
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] 1 new files found on path './mixed.json'
[2026/08/11 14:34:16.731] [debug] [stdout:stdout.0] created event channels: read=36 write=37
[2026/08/11 14:34:16.731] [debug] [router] match rule tail.0:stdout.0
[2026/08/11 14:34:16.731] [debug] [router] match rule tail.1:stdout.0
[2026/08/11 14:34:16.731] [ info] [output:stdout:stdout.0] worker #0 started
[2026/08/11 14:34:16.731] [ info] [sp] stream processor started
[2026/08/11 14:34:16.731] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] [static files] processed 255b
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] [static files] processed 576b
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] inode=1288433329 file=./oneline.json promote to TAIL_EVENT
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] inode=1288433424 file=./mixed.json promote to TAIL_EVENT
[2026/08/11 14:34:16.731] [debug] [input:tail:tail.1] [static files] processed 0b, done
[2026/08/11 14:34:17.733] [debug] [task] created task=0x1025c03a0 id=0 OK
[2026/08/11 14:34:17.733] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
{"date":1786442400.0,"id":1,"level":"info","msg":"oneline record one","file":"./oneline.json"}
{"date":1786442401.0,"id":2,"level":"warn","msg":"oneline record two","file":"./oneline.json"}
{"date":1786442402.0,"id":3,"level":"error","msg":"oneline record three","file":"./oneline.json"}
[2026/08/11 14:34:17.733] [debug] [task] created task=0x1025c0460 id=1 OK
[2026/08/11 14:34:17.733] [debug] [output:stdout:stdout.0] task_id=1 assigned to thread #0
{"date":1786446000.0,"id":101,"level":"info","msg":"single-line record A"}
{"date":1786446001.0,"id":102,"level":"warn","msg":"multiline record B","details":{"user":"alice","action":"login"}}
{"date":1786446002.0,"id":103,"level":"info","msg":"single-line record C"}
{"date":1786446003.0,"id":104,"level":"error","msg":"multiline record D","stack":["frame1","frame2"]}
[2026/08/11 14:34:17.734] [debug] [out flush] cb_destroy coro_id=0
[2026/08/11 14:34:17.734] [debug] [out flush] cb_destroy coro_id=1
[2026/08/11 14:34:17.734] [debug] [task] destroy task=0x1025c03a0 (task_id=0)
[2026/08/11 14:34:17.734] [debug] [task] destroy task=0x1025c0460 (task_id=1)
[2026/08/11 14:34:20.735] [debug] [task] created task=0x77dc00000 id=0 OK
{"date":1786446004.0,"id":105,"level":"info","msg":"single-line record E"}
[2026/08/11 14:34:20.736] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2026/08/11 14:34:20.736] [debug] [out flush] cb_destroy coro_id=2
[2026/08/11 14:34:20.736] [debug] [task] destroy task=0x77dc00000 (task_id=0)
^C[2026/08/11 14:34:27] [engine] caught signal (SIGINT)
[2026/08/11 14:34:27.297] [ info] [input] pausing tail.0
[2026/08/11 14:34:27.297] [ info] [input] pausing tail.1
[2026/08/11 14:34:27.297] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2026/08/11 14:34:27.297] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2026/08/11 14:34:27.297] [debug] [input:tail:tail.1] inode=1288433424 removing file name ./mixed.json
[2026/08/11 14:34:27.297] [debug] [input:tail:tail.0] inode=1288433329 removing file name ./oneline.json

Documentation

Documentation required for this feature. multiline: add built in multiline json parser fluent-bit-docs#2671

Backporting

Backport to latest stable release.

Valgrind output

output

==1640303== Memcheck, a memory error detector
==1640303== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==1640303== Using Valgrind-3.24.0 and LibVEX; rerun with -h for copyright info
==1640303== Command: ./build/bin/flb-it-multiline parser_json
==1640303==
Test parser_json...
----- MULTILINE FLUSH -----
[0] [[1786562895.414402768, {}], {"log"=>"{"id":101,"level":"info","msg":"single-line record A"}
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.484916609, {}], {"log"=>"{
  "id": 102,
  "level": "warn",
  "msg": "multiline record B"
}
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.484939580, {}], {"log"=>"{"id":103,"level":"info","msg":"single-line record C"}
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.486272734, {}], {"log"=>"{
  "id": 104,
  "level": "error",
  "msg": "multiline record D"
}
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.486293395, {}], {"log"=>"{
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.486717910, {}], {"log"=>""bad": true
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.491682517, {}], {"log"=>"{
  "ok": 1
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.491714595, {}], {"log"=>"} trailing
"}]
----------- EOF -----------

----- MULTILINE FLUSH -----
[0] [[1786562895.492978816, {}], {"log"=>"{"id":105,"level":"info","msg":"after boundaries"}
"}]
----------- EOF -----------
==1640303== Warning: invalid file descriptor -1 in syscall close()
[ OK ]
==1640303== Warning: invalid file descriptor -1 in syscall close()
SUCCESS: All unit tests have passed.
==1640303==
==1640303== HEAP SUMMARY:
==1640303==     in use at exit: 0 bytes in 0 blocks
==1640303==   total heap usage: 2,046 allocs, 2,046 frees, 586,762 bytes allocated
==1640303==
==1640303== All heap blocks were freed -- no leaks are possible
==1640303==
==1640303== For lists of detected and suppressed errors, rerun with: -s
==1640303== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Summary by CodeRabbit

New Features
- Added a built-in JSON multiline parser for grouping single-line and pretty-printed JSON records.
- Added json to the supported multiline parser configuration options.
Bug Fixes
- Improved handling of invalid JSON continuation boundaries while preserving subsequent valid records.
Tests
- Added coverage for single-line objects, formatted JSON, flush behavior, and invalid continuations.

coderabbitai · 2026-08-11T19:55:41Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5313dc40-3648-4496-8222-a99f8300ea69

📥 Commits

Reviewing files that changed from the base of the PR and between cc6f17f and 171ea21.

📒 Files selected for processing (2)

src/multiline/flb_ml_parser_json.c
tests/internal/multiline.c

📝 Walkthrough

Walkthrough

The change adds a built-in JSON multiline parser, exposes its factory, documents the json parser name, and adds tests for single-line and pretty-printed JSON records.

Changes

JSON multiline parser support

Layer / File(s)	Summary
Register and expose the JSON parser `include/fluent-bit/multiline/flb_ml_parser.h`, `src/multiline/...`, `plugins/filter_multiline/ml.c`, `plugins/in_tail/tail.c`	The public factory and implementation are added. Built-in creation initializes the parser. Configuration descriptions list `json`.
Validate JSON multiline records `tests/internal/multiline.c`	Fixtures and `test_parser_json` cover single-line and pretty-printed JSON records. The test is registered in `TEST_LIST`.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Config
  participant BuiltinFactory
  participant JSONParser
  participant JSONTest
  Config->>BuiltinFactory: selects json parser
  BuiltinFactory->>JSONParser: creates parser
  JSONTest->>JSONParser: appends fixtures and flushes pending data
  JSONParser-->>JSONTest: emits expected records

Possibly related issues

fluent/fluent-bit#8232: The change implements the requested built-in JSON multiline parser, configuration option, and parser tests.

Suggested reviewers: cosmo0920

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the addition of a built-in JSON multiline parser.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lecaros-json-ml-builtin-parser

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc6f17f9da

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/internal/multiline.c`:
- Around line 367-400: Add invalid-boundary fixtures to json_input and
json_output covering an opening brace followed by an unindented line and a
closing-brace line with trailing content. Assert that each malformed
continuation does not merge into the active JSON group, and include both the
resulting valid records and invalid payload outcomes, including the relevant
boundary-size cases.
- Around line 1229-1230: Restore the flb_time_get(&tm) call immediately before
flb_ml_append_text in the affected test flow, ensuring tm is initialized before
being passed to the append operation and removing the invalid `-.` artifact.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5fe3bb36-a478-4ea5-9d3a-cc31c23d07ad

📥 Commits

Reviewing files that changed from the base of the PR and between ae51533 and cc6f17f.

📒 Files selected for processing (6)

include/fluent-bit/multiline/flb_ml_parser.h
plugins/filter_multiline/ml.c
plugins/in_tail/tail.c
src/multiline/CMakeLists.txt
src/multiline/flb_ml_parser.c
tests/internal/multiline.c

Signed-off-by: lecaros <lecaros@chronosphere.io>

Add flb_ml_parser_json with regex grouping from an opening brace through indented continuation lines. Do not attach a per-line JSON parser; parse the assembled log field with filter_parser instead. Signed-off-by: lecaros <lecaros@chronosphere.io>

Signed-off-by: lecaros <lecaros@chronosphere.io>

Cover unindented continuation lines and closing braces with trailing content so malformed lines are not merged into an active JSON group. Signed-off-by: lecaros <lecaros@chronosphere.io>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 171ea2102a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

lecaros requested review from cosmo0920 and edsiper as code owners August 11, 2026 19:55

github-actions Bot added the docs-required label Aug 11, 2026

lecaros had a problem deploying to pr August 11, 2026 19:56 — with GitHub Actions Failure

chatgpt-codex-connector Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread src/multiline/CMakeLists.txt

Comment thread tests/internal/multiline.c Outdated

coderabbitai Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread tests/internal/multiline.c

Comment thread tests/internal/multiline.c Outdated

lecaros marked this pull request as draft August 11, 2026 19:59

lecaros force-pushed the lecaros-json-ml-builtin-parser branch from cc6f17f to 61b3d44 Compare August 11, 2026 19:59

lecaros temporarily deployed to pr August 11, 2026 20:01 — with GitHub Actions Inactive

lecaros force-pushed the lecaros-json-ml-builtin-parser branch from 61b3d44 to c95af44 Compare August 11, 2026 20:19

lecaros temporarily deployed to pr August 11, 2026 20:19 — with GitHub Actions Inactive

lecaros temporarily deployed to pr August 11, 2026 20:55 — with GitHub Actions Inactive

lecaros temporarily deployed to pr August 11, 2026 20:56 — with GitHub Actions Inactive

lecaros added 4 commits August 12, 2026 10:31

tests: multiline: add tests for json multiline

b624f39

Signed-off-by: lecaros <lecaros@chronosphere.io>

in_tail: add json builtin parser to list

f21887b

Signed-off-by: lecaros <lecaros@chronosphere.io>

filter_multiline: mention json in multiline.parser help

a94e2c8

Signed-off-by: lecaros <lecaros@chronosphere.io>

lecaros force-pushed the lecaros-json-ml-builtin-parser branch from c95af44 to a94e2c8 Compare August 12, 2026 14:38

lecaros temporarily deployed to pr August 12, 2026 14:40 — with GitHub Actions Inactive

lecaros temporarily deployed to pr August 12, 2026 15:11 — with GitHub Actions Inactive

lecaros had a problem deploying to pr August 12, 2026 15:12 — with GitHub Actions Failure

lecaros temporarily deployed to pr August 12, 2026 15:12 — with GitHub Actions Inactive

tests: multiline: add json invalid boundary fixtures

171ea21

Cover unindented continuation lines and closing braces with trailing content so malformed lines are not merged into an active JSON group. Signed-off-by: lecaros <lecaros@chronosphere.io>

lecaros temporarily deployed to pr August 12, 2026 15:33 — with GitHub Actions Inactive

lecaros mentioned this pull request Aug 12, 2026

multiline: add built in multiline json parser fluent/fluent-bit-docs#2671

Open

lecaros marked this pull request as ready for review August 12, 2026 15:46

chatgpt-codex-connector Bot reviewed Aug 12, 2026

View reviewed changes

Comment thread src/multiline/flb_ml_parser_json.c

lecaros temporarily deployed to pr August 12, 2026 16:07 — with GitHub Actions Inactive

lecaros temporarily deployed to pr August 12, 2026 16:11 — with GitHub Actions Inactive

Navigation Menu

Uh oh!

multiline: add built-in json parser for JSON multiline objects - #12264

multiline: add built-in json parser for JSON multiline objects#12264
lecaros wants to merge 5 commits into
masterfrom
lecaros-json-ml-builtin-parser

lecaros commented Aug 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lecaros commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lecaros commented Aug 11, 2026 •

edited

Loading

coderabbitai Bot commented Aug 11, 2026 •

edited

Loading