Yuki Tanaka
Evaluation Engineer · LLM-as-judge, regression suites
Built an eval platform that runs 50K+ judged samples per model release. Calibrated LLM-as-judge against human panels to 0.87 agreement.
Artifact Gallery
Pytest suite of 320 prompt assertions with LLM-as-judge scoring.
Aug 20, 2026Real-time cost monitoring across agent calls with Slack escalation.
Jul 17, 2026Sandboxed tool execution environment with retry + validation.
Jun 14, 2026LangGraph assistant with eval harness for cited literature reviews.
May 11, 2026Streamlit dashboard comparing chunking strategies across 800 eval queries.
Apr 8, 2026Plan-and-execute agent with reflection and tool-retrieval sub-graphs.
Mar 5, 2026Process Trace Summary
- Repository initializedJul 3
Created project scaffold with README and license.
- First commit pushedJul 5
Initial proof-of-concept with placeholder data.
- Evaluation harness addedAug 7
Wired up 50-question regression suite with LLM-as-judge.
- Peer review feedbackAug 9
Two reviewers flagged edge cases in retrieval fallback path.
- Iteration — fallback hardenedSep 11
Added retry + validation; eval score improved 12 points.
- Final submissionSep 13
Artifact submitted for oral defense scheduling.
Oral Defense Transcripts
Recorded Q&A from each verified oral defense session. Expand a session to read the transcript.
Walk us through the architecture of your artifact. Why did you choose this approach?
I chose a plan-and-execute topology because the task required multi-step retrieval with reflection. The plan node decomposes the query, sub-agents retrieve and draft in parallel, and a reflection node scores and routes for a second pass when below threshold.
What evaluation did you run, and what were the headline numbers?
I ran a 50-query regression suite scored by LLM-as-judge calibrated against a human panel (0.86 agreement). Baseline scored 71%; the reflection pass lifted it to 88% with a 14% latency cost, which stayed within budget.
Describe a failure mode you found and how you mitigated it.
Retrieval fallback returned stale context on schema changes. I added a freshness check + retry with a smaller context window, which reduced stale-grounded answers from 9% to under 2%.