Skip to content

recover and use notion sources from skipped slack thread messages #246

@sentry-junior

Description

@sentry-junior

create follow-up for notion source handling when earlier classification skips occur in slack threads

  • reported by <@U039RR91S> after a thread where junior initially answered that it could not verify Sentry's incident priority criteria from public sources, then ignored a later Notion link that contained the policy.
  • thread context showed multiple messages marked assistant skipped: classifier_error, including a direct Notion page URL: https://www.notion.so/sentry/Incident-Management-Policy-f25ff69ef3f24d80aad3ab74ab164e19?...
  • current behavior gap: junior did not recover from the earlier skipped messages and did not use the newly provided Notion source before answering / concluding.
  • likely expected behavior:
    • when a follow-up user message includes a concrete internal doc source like a Notion URL, junior should inspect it if the task is still active or if later asked to explain the miss.
    • classifier skips in intermediate thread messages should not prevent later evidence gathering from thread context.
    • if a prior answer said something was unverified publicly, a subsequent internal source in-thread should trigger re-evaluation instead of staying anchored on the original conclusion.
  • investigation starting points:
    • how thread transcripts are surfaced to the model when some turns are tagged classifier_error
    • whether explicit URLs in skipped messages are omitted from tool-routing or evidence-gathering logic
    • whether Notion skill selection is too dependent on the current turn text and ignores newly surfaced thread context
    • whether follow-up repair behavior should scan the full thread for concrete sources before answering "why didn’t you use X"
  • impact: missed internal source, lower trust, unnecessary manual correction in-channel.

references

  • conversation id: slack:C0AU9QHAVFD:1776889343.944739
  • trace id: dc5b58b9444b2e15679429ffa63f8b0b

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions