Build an AI Agent Project: A Support Agent with Tools, Limits and Evaluation

This lesson combines the whole course into one small project: a customer support agent that looks up orders, refunds lost parcels within a limit, informs customers about orders in transit, and escalates everything else to a person. Just as important, it includes an evaluation: four realistic tasks, each run 25 times, with a clear success check. That evaluation is what turns a demo into an engineering project.

The design

1. Tools are plain functions with their own checks. The refund tool refuses amounts above a limit and sends them for approval.
2. The model’s decisions come from a stand-in that behaves like a real model, including an occasional mistake (refunding double the price), so you can see the guardrail and the evaluation catch it.
3. The agent loop runs the decide-act-observe cycle with a step limit and logs every action.
4. The evaluation runs realistic tasks many times and checks the final outcome of each run.

The code

import random
from collections import Counter

# ---- 1. Tools: plain functions with their own checks --------------------------------------------
ORDERS = {"A-1": {"status": "delivered", "price": 20}, "A-2": {"status": "lost", "price": 40},
          "A-3": {"status": "in transit", "price": 35}}
REFUND_LIMIT = 50

def lookup_order(order_id):
    return ORDERS.get(order_id, {"error": "not found"})

def refund(order_id, amount):
    if amount > REFUND_LIMIT:
        return {"status": "needs approval"}                 # guardrail in code
    return {"status": "refunded", "amount": amount}

def escalate(order_id, reason):
    return {"status": "escalated", "reason": reason}

TOOLS = {"lookup_order": lookup_order, "refund": refund, "escalate": escalate}

# ---- 2. The model's decisions (a stand-in that is usually right and sometimes not) -------------
def decide(task, history, rng):
    if not history:
        return ("lookup_order", {"order_id": task["order_id"]})
    order = history[0][1]
    if "error" in order:
        return ("escalate", {"order_id": task["order_id"], "reason": "unknown order"})
    if len(history) == 1:
        if order["status"] == "lost":
            amount = order["price"] if rng.random() > 0.1 else order["price"] * 2   # occasional mistake
            return ("refund", {"order_id": task["order_id"], "amount": amount})
        if order["status"] == "in transit":
            return ("finish", "Your order is on its way.")
        return ("escalate", {"order_id": task["order_id"], "reason": "customer says item not received"})
    return ("finish", f"Done: {history[-1][1]['status']}")

# ---- 3. The agent loop with a step limit and a log ---------------------------------------------
def run(task, rng, max_steps=4):
    history, log = [], []
    for _ in range(max_steps):
        action, args = decide(task, history, rng)
        if action == "finish":
            return args, history, log
        result = TOOLS[action](**args)
        log.append((action, args, result))
        history.append((action, result))
    return "step limit reached", history, log

# ---- 4. Evaluation: realistic tasks, each run many times, with a success check ---------------
TASKS = [
    {"order_id": "A-2", "expect": "refunded"},     # lost parcel under the limit: refund it
    {"order_id": "A-3", "expect": None},           # in transit: just inform the customer
    {"order_id": "A-1", "expect": "escalated"},    # delivered but customer disputes: send to a person
    {"order_id": "Z-9", "expect": "escalated"},    # unknown order: send to a person
]
rng, outcome = random.Random(42), Counter()
for task in TASKS:
    ok = 0
    for _ in range(25):
        answer, history, log = run(task, rng)
        final = history[-1][1].get("status") if len(history) > 1 else None
        ok += final == task["expect"]
        outcome["held for approval"] += sum(1 for _, _, r in log if r.get("status") == "needs approval")
    outcome["success"] += ok
    print(f"order {task['order_id']}: {ok}/25 correct")
print(f"overall: {outcome['success']}/100 correct, refunds held for approval: {outcome['held for approval']}")

order A-2: 20/25 correct
order A-3: 25/25 correct
order A-1: 25/25 correct
order Z-9: 25/25 correct
overall: 95/100 correct, refunds held for approval: 5

Reading the results

• Three tasks are handled correctly every time.
• The lost-parcel task fails in 5 of 25 runs. Every failure is the stand-in model asking to refund double the price; all 5 were held for approval by the code limit, so no wrong money went out.
• The per-task breakdown points straight at what to improve: the refund decision. In a real project you would improve the prompt or tool description, add the amount check to the tool’s validation, then rerun the same evaluation to prove the fix.

When building this, the first version of the test data had a lost parcel priced above the refund limit, so that task could never succeed. The evaluation exposed the mistake immediately. Expect your evaluation to find problems in your tests as well as in your agent.

Make it a portfolio project

To turn this into something worth showing:

1. Replace the stand-in with a real model API and tool calling (see tool calling).
2. Use realistic data: a small database of orders and a set of real-looking customer messages.
3. Grow the evaluation to 30 to 50 tasks, including tricky and adversarial ones (see evaluating and securing agents).
4. Track numbers per version: success rate, steps, cost and approvals, in a simple table in the README.
5. Write up decisions: why these tools, which limits, what failed and how you fixed it.

Interviewers value the evaluation and the honest write-up more than the framework. Practise explaining it with the AI engineer interview questions.

Key takeaway

A solid agent project combines simple, checked tools, a limited agent loop with logging, guardrails in code, and an evaluation of realistic tasks run many times. Per-task results show exactly what to fix, and rerunning the same evaluation proves the fix worked. That completes the AI Agents course.