multiline: add built in multiline json parser - #2671
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe documentation adds guidance for parsing pretty-printed JSON with Tail’s built-in ChangesJSON multiline parsing
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pipeline/parsers/multiline-parsing.md`:
- Line 869: Update the example reference in the multiline parsing documentation
to point to an existing Fluent Bit repository example path, or add the missing
json-001 example files so the current link resolves successfully.
- Around line 880-884: Add time_keep: on to the YAML json parser definition and
Time_Keep On to the corresponding json_lines parser definition. Update the
output example to retain the parsed time field and include the default date
field emitted by json_lines.
- Line 266: Update the `flush_timeout` description to identify it as the timeout
for a non-terminated multiline buffer waiting for additional input, rather than
framing it only as a shutdown behavior. Keep the `exit_on_eof` and `grace`
recommendation for flushing the final buffer in a separate sentence.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: bc2698d8-2a45-4699-8086-c937b889fe4b
📒 Files selected for processing (3)
pipeline/filters/multiline-stacktrace.mdpipeline/inputs/tail.mdpipeline/parsers/multiline-parsing.md
|
|
||
| - The multiline parser isn't affected by the `buffer_max_size` configuration option, allowing the composed log record to grow beyond this size. The `skip_long_lines` option won't be applied to multiline messages. | ||
| - It's not possible to get the time key from the body of the multiline message. However, it can be extracted and set as a new key by using a filter. | ||
| - When using the built-in `json` parser on static files, the last pretty-printed object may not flush until the next line starting with `{` or until `flush_timeout` (default 4 seconds) during shutdown. With `exit_on_eof`, allow enough `grace` time for the final buffer to flush. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
file="pipeline/parsers/multiline-parsing.md"
printf '%s\n' '--- target context ---'
sed -n '250,275p' "$file"
printf '%s\n' '--- related references ---'
rg -n -C 3 'flush_timeout|exit_on_eof|grace|built-in `json`|static files' "$file"
printf '%s\n' '--- repository references ---'
rg -n -C 3 'flush_timeout|exit_on_eof' --glob '!pipeline/parsers/multiline-parsing.md' .Repository: fluent/fluent-bit-docs
Length of output: 9743
🌐 Web query:
Fluent Bit built-in json parser flush_timeout multiline parser exit_on_eof grace shutdown documentation
💡 Result:
Fluent Bit provides built-in mechanisms for JSON parsing, multiline log aggregation, and lifecycle management. JSON Parser: Fluent Bit includes a built-in JSON parser used to structure log data [1]. You can configure it by setting the format to 'json' in your parser definition [1][2]. Supported parameters include time_key, time_format, and types (e.g., integer, bool, string) to control how fields are decoded [2][3][4]. Multiline Parser: Multiline parsers combine log entries that span multiple lines (such as stack traces) into a single cohesive message [5][6]. They are configured in a multiline_parsers section [7]. You must specify a type (e.g., regex) and define rules consisting of states (start_state, continuation_state) and regex patterns to identify log boundaries [5][7]. Lifecycle and Shutdown Configuration: - grace: Defined in the [SERVICE] section, this sets the duration in seconds that Fluent Bit waits for active tasks to complete before force-quitting during a shutdown process [8][9]. The default is 5 seconds [8]. - flush_timeout: While there is no global flush_timeout, the 'flush' parameter in the [SERVICE] section defines the frequency (in seconds.nanoseconds) with which data is sent to output plugins [8][9]. Within a multiline parser, 'flush_timeout' can also be specified to define how long to wait for a full multiline message before flushing incomplete data [7]. - exit_on_eof: Primarily used with the Tail input plugin, setting 'exit_on_eof' to 'on' causes the plugin to stop reading and trigger an engine shutdown as soon as the end of the input file is reached [10][11]. - Shutdown Behavior: Fluent Bit attempts to synchronize offsets and flush chunks during graceful shutdowns (SIGTERM) [10]. If 'storage.backlog.flush_on_shutdown' is enabled, the engine will also attempt to flush all backlog filesystem chunks before exiting [8].
Citations:
- 1: https://docs.fluentbit.io/manual/data-pipeline/parsers/json
- 2: https://docs.fluentbit.io/manual/data-pipeline/parsers/configuring-parser
- 3: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/parsers-section.md
- 4: https://docs.fluentbit.io/manual/5.0/data-pipeline/parsers/configuring-parser
- 5: https://docs.fluentbit.io/manual/data-pipeline/parsers/multiline-parsing
- 6: https://docs.fluentbit.io/manual/data-pipeline/parsers/multiline-parsing.md
- 7: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/multiline-parsers-section
- 8: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/service-section
- 9: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file.md
- 10: https://docs.fluentbit.io/manual/data-pipeline/inputs/tail
- 11: engine: shutdown hangs at 100% CPU when duplicate STOP signals arrive fluent-bit#11744
🏁 Script executed:
#!/bin/bash
set -e
python3 - <<'PY'
import json
import urllib.request
urls = {
"multiline_json_parser": "https://api.github.com/repos/fluent/fluent-bit/contents/src/multiline",
"issue_12264": "https://api.github.com/repos/fluent/fluent-bit/issues/12264",
}
for name, url in urls.items():
print(f"--- {name} ---")
req = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json", "User-Agent": "review-check"})
with urllib.request.urlopen(req, timeout=20) as response:
data = json.load(response)
if isinstance(data, list):
for item in data:
print(item.get("name"), item.get("download_url"))
else:
print(json.dumps({
"title": data.get("title"),
"state": data.get("state"),
"body": data.get("body"),
"comments": data.get("comments"),
}, indent=2))
PYRepository: fluent/fluent-bit-docs
Length of output: 2477
🌐 Web query:
site:github.com/fluent/fluent-bit "flush_timeout" "json" "exit_on_eof"
💡 Result:
In Fluent Bit, the interaction between flush_timeout (used in multiline parsers) and Exit_On_Eof (used in the tail input plugin) is a known source of data loss during batch processing [1][2][3]. Exit_On_Eof is an input configuration setting that instructs Fluent Bit to terminate as soon as it reaches the end of the monitored file(s) [4][5][6]. It is primarily intended for bulk loading and testing purposes [4][6]. flush_timeout is a configuration option within a multiline parser that defines the maximum amount of time (in milliseconds) the parser will wait for subsequent lines to complete a multiline record before flushing the currently buffered lines [7][8]. The conflict arises because Exit_On_Eof can trigger a shutdown before the multiline parser's flush_timeout interval has elapsed or before pending buffers are processed [1][3]. When Exit_On_Eof is triggered, Fluent Bit initiates its shutdown sequence [5]. If a multiline record is still being buffered (waiting for more lines to arrive or for the flush_timeout to expire), that partial record is often abandoned and lost because the process terminates before the parser can emit the final output [1][3]. Users frequently report that when running in batch mode with Exit_On_Eof, the last multiline log entries are either incomplete or missing entirely [1][2][9]. There is no native configuration option that effectively synchronizes a "safe shutdown" where the parser is forced to flush all remaining multiline buffers before the tail input fully exits [1][3]. Consequently, this combination is generally discouraged for critical batch processing where data integrity is required [3].
Citations:
- 1: Multiline Parser adds support for end_state and start flushing fluent-bit#10094
- 2: How to read multiline json format fluent-bit#2418
- 3: exit_on_eof wait until output flushes all the buffers fluent-bit#11193
- 4: https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/inputs/tail.md
- 5: https://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail.c
- 6: in_tail: Add Exit_On_EOF option to tail input fluent-bit#878
- 7: https://github.com/fluent/fluent-bit-docs/blob/master/administration/configuring-fluent-bit/yaml/multiline-parsers-section.md
- 8: flush_timeout of multiline parser does not reset the state machine fluent-bit#9109
- 9: multiline parser doens't play well with the last record fluent-bit#8623
🏁 Script executed:
#!/bin/bash
set -e
api='https://api.github.com'
hdr=(-H 'Accept: application/vnd.github+json' -H 'User-Agent: review-check')
printf '%s\n' '--- pull request 12264 ---'
curl -ksS "${hdr[@]}" "$api/repos/fluent/fluent-bit/pulls/12264" |
jq '{title,body,state,merged_at,changed_files}'
printf '%s\n' '--- pull request files ---'
curl -ksS "${hdr[@]}" "$api/repos/fluent/fluent-bit/pulls/12264/files?per_page=100" |
jq -r '.[] | [.filename, .status, (.patch // "")] | `@tsv`'
printf '%s\n' '--- multiline source candidates ---'
curl -ksS "${hdr[@]}" "$api/repos/fluent/fluent-bit/contents/src" |
jq -r '.[].name' | grep -Ei 'multi|parser'
printf '%s\n' '--- tail source references ---'
curl -ksS "${hdr[@]}" "$api/repos/fluent/fluent-bit/git/trees/master?recursive=1" |
jq -r '.tree[] | select(.path | test("multiline|in_tail|parser"; "i")) | .path' |
grep -E '(^src/|^plugins/in_tail/)' | head -100Repository: fluent/fluent-bit-docs
Length of output: 382
Describe flush_timeout as a multiline-buffer timeout. It applies when a non-terminated buffer waits for more input, not only during shutdown. Keep the exit_on_eof and grace guidance in a separate sentence.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pipeline/parsers/multiline-parsing.md` at line 266, Update the
`flush_timeout` description to identify it as the timeout for a non-terminated
multiline buffer waiting for additional input, rather than framing it only as a
shutdown behavior. Keep the `exit_on_eof` and `grace` recommendation for
flushing the final buffer in a separate sentence.
| parsers: | ||
| - name: json | ||
| format: json | ||
| time_key: time | ||
| time_format: "%Y-%m-%dT%H:%M:%S.%L" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target file locations ---'
rg -n -C 12 'time_key|time_keep|json_lines|date|github.com/fluent/fluent-bit-docs|Time_Keep|Time_Key' pipeline/parsers/multiline-parsing.md
printf '%s\n' '--- repository references to JSON parser timestamp behavior ---'
rg -n -S -C 4 'time_keep|Time_Keep|json_lines|Time_Key|generated.*date|date.*field' . \
-g '*.md' -g '*.yaml' -g '*.yml' -g '*.conf' -g '*.c' -g '*.cc' -g '*.cpp' -g '*.h' -g '*.cxx' \
| head -n 500Repository: fluent/fluent-bit-docs
Length of output: 40264
🌐 Web query:
Fluent Bit JSON parser Time_Keep default date field json_lines multiline parser documentation
💡 Result:
In Fluent Bit, the Time_Keep configuration option determines whether the original time field is retained after it has been parsed and used to set the event's internal timestamp [1][2]. When Time_Keep is enabled (set to 'on' or 'true'), the parser keeps the original time key in the resulting record [1][2]. When it is disabled (the default setting), the parser drops the original time field once the timestamp has been processed [1][2]. This setting applies to custom parsers, including those using the JSON format [1][3][2]. Regarding your query about multiline parsing and 'json_lines': 1. Multiline Parsing: Fluent Bit handles multiline logs using a separate multiline parser configuration, which can be defined in a [MULTILINE_PARSER] section (or under a multiline_parsers section in YAML) [4][5]. 2. JSON Lines: If you are processing a file where each line is a valid JSON object (often referred to as JSON Lines), you typically do not need a complex multiline parser. Instead, you can use the Tail input plugin with the 'json' parser directly [6][7]. If your logs are multiline JSON objects, the multiline parser must be configured to identify the start of a record and can optionally apply a pre-defined parser to that content [4][5]. To ensure the time field is kept when using a JSON parser, you would configure your parser as follows: Example YAML configuration: parsers: - name: my_json_parser format: json time_key: time time_format: '%Y-%m-%dT%H:%M:%S.%L' time_keep: on Example classic configuration: [PARSER] Name my_json_parser Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On
Citations:
- 1: https://docs.fluentbit.io/manual/data-pipeline/parsers/configuring-parser
- 2: https://docs.fluentbit.io/manual/4.0/data-pipeline/parsers/configuring-parser
- 3: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/parsers-section
- 4: https://docs.fluentbit.io/manual/data-pipeline/parsers/multiline-parsing
- 5: https://docs.fluentbit.io/manual/4.1/administration/configuring-fluent-bit/multiline-parsing
- 6: https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/parsers/json.md
- 7: https://docs.fluentbit.io/manual/data-pipeline/parsers/json
🏁 Script executed:
#!/bin/bash
set -eu
sed -n '946,990p' pipeline/parsers/multiline-parsing.md
printf '%s\n' '--- link target checks ---'
if command -v git >/dev/null 2>&1; then
git ls-remote https://github.com/fluent/fluent-bit.git \
refs/heads/master refs/heads/main 2>/dev/null | head -n 5 || true
fi
printf '%s\n' '--- local documentation for parser retention and stdout date fields ---'
sed -n '1,35p' pipeline/parsers/configuring-parser.md
sed -n '1,28p' pipeline/outputs/standard-output.mdRepository: fluent/fluent-bit-docs
Length of output: 6677
Keep the parsed time field in both parser definitions.
time_keep defaults to false, so the parser removes time. Add time_keep: on and Time_Keep On. Update the output example to include both time and the default date field emitted by json_lines.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pipeline/parsers/multiline-parsing.md` around lines 880 - 884, Add time_keep:
on to the YAML json parser definition and Time_Keep On to the corresponding
json_lines parser definition. Update the output example to retain the parsed
time field and include the default date field emitted by json_lines.
Signed-off-by: lecaros <lecaros@chronosphere.io>
f190973 to
12f0308
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| | `docker` | Process a log entry generated by a Docker container engine. This parser supports the concatenation of large log entries split by Docker. If you use this parser, and you also want to concatenate log lines like stack traces, you can add the [multiline filter](../../pipeline/filters/multiline-stacktrace.md) to specify additional parsers. | | ||
| | `go` | Process log entries generated by a Go-based language application and perform concatenation if multiline messages are detected. | | ||
| | `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | ||
| | `json` | Group pretty-printed JSON objects that span multiple lines. The parser concatenates lines only; use a [Parser filter](../../pipeline/filters/parser.md) with `parser: json` on the assembled `log` field to extract structured fields. | |
There was a problem hiding this comment.
[markdownlint] reported by reviewdog 🐶
error MD060/table-column-style Table column style [Table pipe has extra space to the left for style "compact"]
| | `go` | Process log entries generated by a Go-based language application and perform concatenation if multiline messages are detected. | | ||
| | `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | ||
| | `json` | Group pretty-printed JSON objects that span multiple lines. The parser concatenates lines only; use a [Parser filter](../../pipeline/filters/parser.md) with `parser: json` on the assembled `log` field to extract structured fields. | | ||
| | `python` | Process log entries generated by a Python-based language application and perform concatenation if multiline messages are detected. | | ||
| | `ruby` | Process log entries generated by a Ruby-based language application and perform concatenation if multiline messages are detected. | |
There was a problem hiding this comment.
[markdownlint-fix] reported by reviewdog 🐶
| | `go` | Process log entries generated by a Go-based language application and perform concatenation if multiline messages are detected. | | |
| | `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | |
| | `json` | Group pretty-printed JSON objects that span multiple lines. The parser concatenates lines only; use a [Parser filter](../../pipeline/filters/parser.md) with `parser: json` on the assembled `log` field to extract structured fields. | | |
| | `python` | Process log entries generated by a Python-based language application and perform concatenation if multiline messages are detected. | | |
| | `ruby` | Process log entries generated by a Ruby-based language application and perform concatenation if multiline messages are detected. | | |
| | `go` | Process log entries generated by a Go-based language application and perform concatenation if multiline messages are detected. | | |
| | `java` | Process log entries generated by a Google Cloud Java language application and perform concatenation if multiline messages are detected. | | |
| | `json` | Group pretty-printed JSON objects that span multiple lines. The parser concatenates lines only; use a [Parser filter](../../pipeline/filters/parser.md) with `parser: json` on the assembled `log` field to extract structured fields. | | |
| | `python` | Process log entries generated by a Python-based language application and perform concatenation if multiline messages are detected. | | |
| | `ruby` | Process log entries generated by a Ruby-based language application and perform concatenation if multiline messages are detected. | |
1 participant
adds documentation for fluent/fluent-bit#12264
Summary by CodeRabbit