Tag: claude

Posted on

Using smaller/less capable local LLMs together with Claude models for a ‘second opinion’

I want to use a Qwen 3.6 local model but have found from testing with various harnesses it is prone to overthinking and goes in circles on some tasks. To avoid this I want to use Qwen as my default/primary model but then spawn a second sub-agent using a more capable Claude model to provide guidance on the solution.

I already have llama.cpp running a server in router mode to run my local models, I asked Claude what approach to use to configure the default model and sub-agent model approach and it recommended to configure a Claude Code agent that passes requests through a LiteLLM proxy to direct requests either to the local model or to cloud based Claude.

Sub-agent approach using LiteLLM

Note this approach requires a Claude subscription with access to API keys. My Pro subscription doesn’t include this, so I couldn’t use this approach, see the following skill only approach that follows this section.

LiteLLM config:

# litellm_config.yaml
model_list:
  - model_name: qwen3.6
    litellm_params:
      model: openai/qwen3.6-35b
      api_base: http://host.docker.internal:8090/v1

  - model_name: claude-opus-5-5
    litellm_params:
      model: anthropic/claude-opus-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

From the LiteLLM docs here, startup a Docker container with the above config:

docker run
-d
--name litellm
--mount type=bind,source="$PWD/litellm_config.yaml",target=/app/config.yaml,readonly
-e LITELLM_MASTER_KEY=[your-local-litellm-key-here]
--add-host=host.docker.internal:host-gateway
-p 4000:4000
docker.litellm.ai/berriai/litellm:latest
--config /app/config.yaml

Point Claude Code at the proxy. Qwen becomes the default model and the opus alias still reaches real Claude:

export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_AUTH_TOKEN=<litellm master key from abovw>
export ANTHROPIC_MODEL=qwen3.6
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen3.6 # keep background calls local
export ANTHROPIC_DEFAULT_OPUS_MODEL=claude-opus-5-5 # the escalation target

Claude’s suggestion on the agent config – add a second-opinion subagent in .claude/agents/second-opinion.md::agents

---
name: second-opinion
description: Use when stuck — same error after 2 fix attempts, torn between approaches, or reasoning is repeating itself. Pass the goal, what was tried, and the exact error text.
model: opus
tools: Read, Grep, Glob
---
You are a senior reviewer consulted by a smaller model that is stuck.
Read the relevant files yourself. Reply briefly with: (1) the diagnosis, (2) the single next step, (3) what to stop doing.

Giving it read-only tools keeps Claude in an advisory role. It can check the code itself instead of relying only on Qwen's summary, and it can't take over the work, which keeps your API spend down.

Claude’s recommendation for the agent trigger and a hook:

Trigger:

Set explicit triggers in CLAUDE.md, for example: "If a fix fails twice, or you are about to re-try something you've already tried, call the second-opinion agent before continuing." Concrete rules like these work much better than "when unsure."

Hook:

Add a hook so the call-out happens even when Qwen misses it. Use a PostToolUse hook on Bash that counts consecutive failures, or repeated edits to the same file, in a state file. At a threshold such as 3, it returns additionalContext telling the model to call second-opinion now. The hook detects the loop; Claude only gets called when it's needed.to

Simpler skill only approach

After trying the agent and LiteLLM approach above, I found out that with a Claude Pro account you can’t create API keys, so the LiteLLM config that references os.environ/ANTHROPIC_API_KEY won’t work. Instead, change the agent config to a Claude skill instead. With the approach you don’t also don’t need LiteLLM, point Claude directly to your llama.cpp URL:

export ANTHROPIC_BASE_URL=http://localhost:8090
export ANTHROPIC_AUTH_TOKEN=dummy # llama-server ignores it unless started with --api-key
export ANTHROPIC_MODEL=qwen/qwen3.6-35b-a3b-agent # router selects the model by this alias
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen/qwen3.6-35b-a3b-agent # keep background calls local
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 # matches llama-server ctx-size

# ArtifactData has a nested maxLength llama.cpp cannot turn into a grammar
claude --disallowedTools=ArtifactData "$@"

Create this skill in .claude/skills/ that tells the default agent to write to a second-opinion.md file that Claude can pick up:

---
name: second-opinion
description: Ask Claude for a second opinion when stuck. Use when a fix has failed twice, when you are about to retry something already tried, when the same error keeps coming back, or when you are going back and forth between approaches
---
# Second opinion from Claude

Use this as soon as a trigger applies. Do not attempt another fix first.

## Steps

1. Write a brief to `.claude/second-opinion-brief.md` (overwrite it) with these sections:
   - **Goal**: what you are trying to achieve, in one or two sentences.
   - **Relevant files**: paths Claude should read.
   - **Tried so far**: each attempt and what happened.
   - **Current error**: the exact error text or failing output, copied verbatim.
   - **Question**: what you need decided.

2. From the project root, run this with the Bash tool, setting `timeout` to `600000`:

   ```
   ~/.claude/skills/second-opinion/ask-claude.sh
   ```

3. Follow the **Next step** in the reply and stop doing whatever the reply lists under **Stop doing**. Do not ask again for the same problem unless the sugges>

If the script fails with a login or authentication error, tell the user to run `claude` outside the LiteLLM launcher and `/login` with their Claude Pro accoun>

For the ask-claude.sh script as part of the skill:

#!/usr/bin/env bash
set -euo pipefail

project_dir="$PWD"
brief_file="${1:-$project_dir/.claude/second-opinion-brief.md}"

if [[ ! -s "$brief_file" ]]; then
  echo "error: brief file '$brief_file' is missing or empty" >&2
  exit 2
fi

system_prompt='You are a senior engineer giving a second opinion to a smaller local model that is stuck.
Read the relevant project files yourself rather than trusting the brief alone.
Reply in under 250 words with exactly three sections:
1. Diagnosis - the most likely root cause.
2. Next step - one concrete action, with file paths and code if needed.
3. Stop doing - what the model should abandon.'

cd "$project_dir"
exec claude -p \
  --model "${SECOND_OPINION_MODEL:-opus}" \
  --tools "Read,Grep,Glob" \
  --no-session-persistence \
  --append-system-prompt "$system_prompt" \
  < "$brief_file"

In your CLAUDE.md for your project, or globally, define rules for when your primary local model should invoke the second-opinion skill:

# Triggers

If the same error message appears in two test runs, or you are about to re-try something you've already tried, your next action must be the Skill tool with second-opinion. Changing a config key's name or location counts as the same fix. If the suggested step fails, ask again.

So far with Qwen 3.6 I’ve found it’s interpretation of ‘the same error message in two test runs’ is a bit hit and miss. It’s also not particularly reliable on counting occurrences of errors. These rules could be tightened up, and depending on what you’re working on, the wording of the trigger could be made more specific to exactly the types of errors you’re expecting, either error codes or messages. That said, if I find Qwen taking too long trying alternative approaches, you can just prompt it manually to use the second-opinion skill and it works as you’d expect – it summarises the currently tried approaches in the input file and then feeds it into to a prompt with Claude.

Proudly powered by WordPress