feat(aws_kinesis_firehose source): include attributes from X-Amz-Firehose-Common-Attributes header in the log event

[tchanturia]

Summary

Adds support for including attributes from the X-Amz-Firehose-Common-Attributes header in the log events for the aws_kinesis_firehose source.

Configurable via the new common_attributes config property similar to headers in http_server. Wildcard * includes all available common attributes in the request header.

The attributes are inserted as an object map to the metadata into .common_attributes property, making it also easy to merge all passed attributes into, for example, tags for Datadog sink.

Vector configuration

Source configuration variants

No `common_attributes:

sources:
  kinesis_firehose:
    type: aws_kinesis_firehose
    address: 0.0.0.0:3000
    access_keys:
      - SECRET[vector_secrets.firehose_access_key]
    store_access_key: false

Wildcard:

sources:
  kinesis_firehose:
    type: aws_kinesis_firehose
    address: 0.0.0.0:3000
    access_keys:
      - SECRET[vector_secrets.firehose_access_key]
    store_access_key: false
    common_attributes:
      - "*"

Specific common_attributes:

sources:
  kinesis_firehose:
    type: aws_kinesis_firehose
    address: 0.0.0.0:3000
    access_keys:
      - SECRET[vector_secrets.firehose_access_key]
    store_access_key: false
    common_attributes:
      - "environment"
      - "appgroup"
      - "absent_attribute"

Console sink:

sinks:
  console:
    inputs: [kinesis_firehose]
    type: console
    encoding:
      codec: json

How did you test this PR?

• Unit tests
• Manual tests in AWS

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Closes: #24913

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[tchanturia]

I have read the CLA Document and I hereby sign the CLA

[pront]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".