Nextcraft
Back to build
Assessment · Defense

RAG Pipeline Design

AI Orchestration Engineer · oral defense and rubric review

Submitted Artifact

import { ChatOpenAI } from '@langchain/openai';
import { tool } from '@langchain/core/tools';
import { z } from 'zod';

const getWeather = tool(
  async ({ city, units }) => {
    const res = await fetch(`/api/weather?city=${city}&units=${units}`);
    return res.json();
  },
  {
    name: 'get_weather',
    description: 'Fetch the current weather for a city',
    schema: z.object({
      city: z.string(),
      units: z.enum(['celsius', 'fahrenheit']).default('celsius'),
    }),
  },
);

export async function main(query: string) {
  const model = new ChatOpenAI({ model: 'gpt-4o-mini' });
  const modelWithTools = model.bindTools([getWeather]);
  const response = await modelWithTools.invoke(query);
  return response.tool_calls;
}

Assessment rubric

Criteria and weights

Correctness of agent architecture
25%
Tool schema design & validation
20%
Error handling & fallbacks
20%
Test coverage
15%
Code clarity & documentation
20%

AI Assessor

Automated review results

84

Overall score

Pass threshold: 75 · Passing

  • Correctness of agent architecture
    92
  • Tool schema design & validation
    88
  • Error handling & fallbacks
    61
  • Test coverage
    84
  • Code clarity & documentation
    90

Feedback

Strong tool-schema design and clear architecture. Error handling loses points: malformed model output is not guarded with a fallback parser. Add a retry with a structured-output schema and re-run the eval harness before your oral defense.

Process trace

Build history captured during the sandbox session

  1. File created: src/main.ts14:02:11
  2. First build attempt (failed)14:09:48
  3. Test suite passed (2/2)14:12:30
  4. Commit: scaffold agent entrypoint14:18:05
  5. Tool schema validated14:21:42
  6. Build attempt 2 (success)14:25:17
  7. Commit: implement tool calling14:28:03
  8. Artifact submitted for assessment14:31:50

Oral defense

Live oral exam with an AI examiner

Start Oral Defense

Defense transcript

AI

Examiner

Walk us through your design choices for multi-agent communication in this artifact. Why a blackboard architecture over direct message passing?

AR

Learner

I chose a shared blackboard because the agents publish partial results that others consume asynchronously — direct messaging would have tightly coupled them and made re-planning harder. The blackboard also gives me a clean audit trail for each step.

AI

Examiner

What failure mode did you observe under load, and how did you mitigate it?

AR

Learner

At 50 concurrent requests the planner became a bottleneck because every agent waited on a fresh plan. I added a plan cache keyed by intent signature and moved re-planning to a debounce — throughput improved 3x with no measurable quality regression.