Structured log tips/workflow

[bisoldi]

What were you initially searching for in the docs?
More information how JSON / structured logging can enhance our ability to process logs and make use of them, especially when troubleshooting a bug and what the best practices are to utilize structured logging.

Is this related to an existing part of the documentation? Please share a link
N/A

Describe how we could make it clearer
Highlighting and elaborating on the following would be a good start:

1. Structured Logs allow you to have a contract of what’s going to be available

• What datapoints are most important for inclusion in the logs?
• What datapoints are important for inclusion in the logs when the Lambda is a part of a Step Function and therefore one of several functions?

2. All Log analytics tool support indexing JSON natively - this means you can search logs more efficiently, and discover what keys are available

• Perhaps some references would be valuable here, especially if CloudWatch Log ingestion is native to the log analytic tool

3. CloudWatch Logs Insights allow you to create graphs more easily if your Logs are in JSON format

• Perhaps some references would be valuable here as well, especially if CloudWatch Log ingestion is native to the graphing tool

4. What are the recommended workflows when responding to an error / bug?

• Is CloudWatch Logs still meant to be a UI for reading logs?
• How does Insights factor in?

If you have a proposed update, please share it here

[heitorlessa]

Hey Brooks - Any suggestions are to where this might best fit?

A new section in the docs? A Lab, perhaps?

[bisoldi]

Sorry for the delay...I think adopting a strategy of showing how adding structure to logs, and combining them with metrics is better requires both labs and a dedicated section in the documents. Hell, it probably requires a book and I predict it will soon evolve into its own ecosystem of libraries, tooling and applications.

I think it will be tedious, but in the long run, invaluable for generating adoption as we evolve from a centralized "Log everything, from everywhere, in whatever format (typically unstructured) to one target" methodology to a bit more of a decentralized "Log everything, from everywhere, with standardization, combining metrics, and structured and unstructured information" approach.

Maybe start with the following:

Documentation:

1. Discussion of WHY structured logging. What problems does it solve? What new problems does it potentially cause? What is the paradigm shift in working with structured logging over contemporary logging.

2. Coverage for Feature: Integrate custom metrics with new CloudWatch embedded metric #1 in my first post (structured log contract) with documentation of the datapoints most valuable in a variety of processing environments (Lambda, ECS/Fargate, EC2, EMR, Beanstalk, Processing invoked by a Step Function).

3. Documentation of best practices for structured logging, to include discussion of the "best" datapoints to include in every single log message vs. datapoints to include once when logging at the beginning of each invocation/process.

4. Discussion of the workflows and best practices that will best enable devs and customer support staff to use the new logging capability to more easily and quickly both troubleshoot and discover problems. I understand this will probably bleed into discussions of non-AWS tools (I'm not sure if that would be allowed) and will be time-consuming, so maybe this is high-level and broad discussion of what will eventually turn into labs.

Labs:

1. Equally important to showing vs. explaining.

2. Hands-on labs (ala "Wild Rydes") in multiple different environments producing structured logs, following best practices of when and which datapoints to include.

3. Hands-on labs teaching how to provision and use services and/or 3rd party (preferably open sourced) tools to enable easy, and rapid searching and discovery. What are the workflows for making use of the structured logs produced in step Question: No module named 'aws_lambda_logging' #2 above?

Again, I know this is a LOT and warrants a book. If this is too much, maybe some lite version of the above, or a full version of the above, but focusing on one environment (e.g. Lambda). I, selfishly, would want to see Lambda + Step Functions covered.

Does this help / answer the question?

Thanks!

[heitorlessa]

Thanks a lot @bisoldi - It does help, as this is a much bigger domain than Powertools alone per se -- After 1.7.0, I'll regroup with @nmoutschen and @cakepietoast to think this through. I'd guess it'd have to be a mix of Blog posts, Labs, and a Webinar-like to cover this, and to adapt different types of learners to multiple operational aspects.