Pull ACA console logs into Airflow via Log Analytics - #34
Merged
Conversation
After ingest/enrich ACA jobs finish, query ContainerAppConsoleLogs_CL in log-fp-<team>, filter to application loggers (src.* / pipeline), and emit the lines in the Airflow task log. Poll every 5s, cap wait at 3 min. Pass TEAM from pipeline_dag so the workspace resolves per team. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I built
After ACA ingest/enrich jobs finish, Airflow pulls container stdout from the team's Log Analytics workspace (
log-fp-<team>) and replays application log lines in the task log. Azure SDK noise is filtered out.Why this approach
With
INGEST_MODE=aca, container output never reaches Airflow directly — it lands inContainerAppConsoleLogs_CL. Querying Log Analytics after the job completes is the same path operators use in the portal, without adding dependencies to the Airflow image. Polling every 5s (max 3 min) was verified on the teacher sandbox.Contract impact
None
How to run
cd data uv run pytest tests/common/test_aca.pyAfter merge, trigger
final_project_pipeline_devorfinal_project_pipelineon the team VM. The ingest task log should include a--- console log for job-ingest-... ---block withsrc.*/pipelinelines.Self-check