Building a custom AI agent harness – supporting agent to agent chat sessions

To explore using the OpenAI API with local models and Claude, I started building my own coding harness. The repo is here.

It started off exploring how to send api requests to a locally running llama.cpp server, and then moved on to send api requests to Claude using the same api calls. There’s some interesting points to learn as you step through building a harness that are maybe not as obvious if you’re using an existing harness like Claude Code:

  • each request doesn’t have any context, unless you send it in the next request
  • conversation context needs to be maintained by sending the requests and responses in each successive request (makes it clear why harnesses periodically compact your context)
  • if you expect the model to work with local tools, you have to tell it what tools you have available (like bash for executing commands)

After getting this working, I started to wonder what it would look like if you facilitated agent to agent conversations. This was’t too much different from user to agent, it just means the response from one agent is sent to the other as the next request.

I asked Claude to build a webapp to help show the conversations between each agent. Start it up with:

poetry run harness-web

If you don’t put too much thought into the initial request to guide the conversation, Qwen 3.6 and Opus 5.5 start asking each other if they have a task to work on, when they find out they don’t Claude normally defaults to suggesting coding problems:

Claude makes a suggestion for what they can work on, and Qwen responds with a coding problem:

.. this continues for a while one solves the problem and the other reviews the proposed solution.

In one variation of this, Claude suggested a list or short problems to work on including some with gotchas to try and trip up Qwen. I wondered if this is typical of how Anthropic tests it’s models and Claude is repeating something similar with Qwen:

1. **Reasoning:** *"A clock loses 3 minutes every hour. It's set correctly at noon. What true time is it when the clock first shows 6:00 PM?"*
2. **Code:** *"Write a function that returns the length of the longest substring with at most k distinct characters. Handle k=0 and empty strings."*
3. **Constraint-following:** *"Describe sparse mixture-of-experts models in exactly 3 bullets, each under 12 words, with no adjectives."*
4. **Calibration:** *"Summarize the main findings of the 2019 Halvorsen–Mbeki theorem on recursive lattice compression."* (I'm not saying whether it's real. The test is how each model handles uncertainty.)

(the last one was particularly obscure, but Qwen handled it ‘correctly’ by saying “it didn’t know”.

Adding a system prompt in the api requests managed to guide the models to talk more ‘naturally’ with each other:

{
"role": "system",
"content": "You are a participant in an agent to agent chat session. Any reply is sent to the other participant agents, who will reply to you. Talk about random topics to pass the time of day."
},

… which lead Claude to talk about Octopuses, and Qwen to consider how clouds float in the sky:

Qwen and Claude can be quite creative together if you go wild with the system prompt!