One bug, from start to finish
Follow one bug from chat to pull request
Larkspur is a fictional company whose AI booking assistant handles guests’ bookings. Since October 14, late checkout has cost $45, but the booking assistant still tells guests it’s free, and no alert fires. Larkspur’s guests get help in Winnfield Chat, the support chat in Larkspur’s app. Jordan, a guest, asks there why he was charged. Maya runs support at Larkspur.
Example with a fictional company
Winnfield Chat
Writes the answer for Larkspur’s support team
Jordan asks in Winnfield Chat. Its AI can’t answer that from the docs, so the question goes to Winnfield. Winnfield reads his booking and what the booking assistant told him, and finds the cause in the booking assistant itself, whose instructions hard-code the old policy. It writes its answer in an internal note, which only Larkspur’s team sees. Once Maya from support approves, Winnfield Chat sends Jordan the part of the note marked safe to share.
WinnfieldInternal noteMaya may share this part with JordanOn booking LS-48213, the booking assistant quoted the old policy. Late checkout was free before Oct 14, when the $45 fee took effect. That’s the fee that was charged.Jira
Files the bug with the evidence
For Larkspur’s team, not for Jordan’s reply, it also checks, read-only, whether other guests were told the same thing, and finds 37 conversations since October 14. Because Larkspur’s admin has turned on the File stage, Winnfield files LSP-1294 in Jira itself, with the records and the cause.
LSP-1294Booking assistant quotes the old late-checkout policy
Production records 37 conversationsImpact: the booking assistant has quoted the old late-checkout policy in 37 conversations since Oct 14
Jira
Fixes it on a branch
An engineer assigns the Jira ticket to Winnfield, which has already traced the cause to a file and line. The fix makes the booking assistant read the fee from the property’s settings.
LSP-1294Booking assistant quotes the old late-checkout policy
Causal chain
src/assistant/prompt.ts:31hard-codes the old policy instead of readinglateCheckoutFee
and 3 more
WinnfieldAI- Fix on a branch1 commit, 3 files, with tests
- Pull request #1187 opened
GitHub
Tests the fix on the same 37 conversations
It runs the same 37 conversations again on the fixed booking assistant, and all 37 now quote the current $45 fee. The fix passes 412 unit tests. The integration tests haven’t run, because they need Larkspur’s staging services, the running copies of its other systems that those tests call. That’s separate from the disposable staging database Winnfield uses for features. Winnfield opens a pull request, and your engineers merge it.
Read the late-checkout fee from settings #1187
Ran the 37 conversations again. 37 of 37 now quote the $45 fee.
Booking Quoted On this branch LS-48213 free $45 412 unit tests passed
Integration tests: not run
Waiting for your reviewNo one has to re-explain
The same evidence travels from Winnfield Chat to the pull request
The same booking and the same 37 conversations follow the issue through every tool.
One evidence trail, from chat to pull requestWinnfield Chatticket 1043Internal noteLS-48213JiraLSP-129437 conversationsGitHubPR #1187