model-tracking-protocol

What this skill does

# Model Tracking Protocol

**Version:** 1.0.0
**Purpose:** MANDATORY tracking protocol for multi-model validation to prevent incomplete reviews
**Status:** Production Ready

## Overview

This skill defines the MANDATORY tracking protocol for multi-model validation. It provides templates and procedures that make proper tracking unforgettable.

**The Problem This Solves:**

Agents often launch multiple external AI models but fail to:
- Create structured tracking tables before launch
- Collect timing and performance data during execution
- Document failures with error messages
- Perform consensus analysis comparing model findings
- Present results in a structured format

**The Solution:**

This skill provides MANDATORY checklists, templates, and protocols that ensure complete tracking. Missing ANY of these steps = INCOMPLETE review.

---

## Table of Contents

1. [MANDATORY Pre-Launch Checklist](#mandatory-pre-launch-checklist)
2. [Tracking Table Templates](#tracking-table-templates)
3. [Per-Model Status Updates](#per-model-status-updates)
4. [Failure Documentation Protocol](#failure-documentation-protocol)
5. [Consensus Analysis Requirements](#consensus-analysis-requirements)
6. [Results Presentation Template](#results-presentation-template)
7. [Common Failures and Prevention](#common-failures-and-prevention)
8. [Integration Examples](#integration-examples)

---

## MANDATORY Pre-Launch Checklist

**You MUST complete ALL items before launching ANY external models.**

This is NOT optional. If you skip this, your multi-model validation is INCOMPLETE.

### Checklist (Copy and Complete)

```
PRE-LAUNCH VERIFICATION (complete before Task calls):

[ ] 1. SESSION_ID created: ________________________
[ ] 2. SESSION_DIR created: ________________________
[ ] 3. Tracking table written to: $SESSION_DIR/tracking.md
[ ] 4. Start time recorded: SESSION_START=$(date +%s)
[ ] 5. Model list confirmed (comma-separated): ________________________
[ ] 6. Per-model timing arrays initialized
[ ] 7. Code context written to session directory
[ ] 8. Tracking marker created: /tmp/.claude-multi-model-active

If ANY item is unchecked, STOP and complete it before proceeding.
```

### Why Pre-Launch Matters

Without pre-launch setup, you will:
- Lose timing data (cannot calculate speed accurately)
- Miss failed model details (no structured place to record)
- Skip consensus analysis (no model list to compare)
- Present incomplete results (no tracking table to populate)

### Pre-Launch Script Template

**CRITICAL CONSENSUS FIX APPLIED:** Use file-based detection instead of environment variables.

```bash
#!/bin/bash
# Run this BEFORE launching any Task calls

# 1. Create unique session
SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="/tmp/${SESSION_ID}"
mkdir -p "$SESSION_DIR"

# 2. Record start time
SESSION_START=$(date +%s)

# 3. Create tracking table
cat > "$SESSION_DIR/tracking.md" << EOF
# Multi-Model Tracking

## Session Info
- Session ID: ${SESSION_ID}
- Started: $(date -u +%Y-%m-%dT%H:%M:%SZ)
- Models Requested: [FILL]

## Model Status

| Model | Agent ID | Status | Start | End | Duration | Issues | Quality | Notes |
|-------|----------|--------|-------|-----|----------|--------|---------|-------|
| [MODEL 1] | | pending | | | | | | |
| [MODEL 2] | | pending | | | | | | |
| [MODEL 3] | | pending | | | | | | |

## Failures

| Model | Failure Type | Error Message | Retry? |
|-------|--------------|---------------|--------|

## Consensus

| Issue | Model 1 | Model 2 | Model 3 | Agreement |
|-------|---------|---------|---------|-----------|

EOF

# 4. Initialize timing arrays
declare -A MODEL_START_TIMES
declare -A MODEL_END_TIMES
declare -A MODEL_STATUS

# 5. Create tracking marker file (CRITICAL FIX)
# This allows hooks to detect that tracking is active
echo "$SESSION_DIR" > /tmp/.claude-multi-model-active

echo "Pre-launch setup complete. Session: $SESSION_ID"
echo "Directory: $SESSION_DIR"
echo "Tracking table: $SESSION_DIR/tracking.md"
```

### Strict Mode (Optional)

For stricter enforcement, set:

```bash
export CLAUDE_STRICT_TRACKING=true
```

When enabled, hooks will BLOCK execution if tracking is not set up, rather than just warning.

---

## Tracking Table Templates

### Template A: Simple Model Tracking (3-5 models)

```markdown
| Model | Status | Time | Issues | Quality | Cost |
|-------|--------|------|--------|---------|------|
| claude-embedded | pending | - | - | - | FREE |
| x-ai/grok-code-fast-1 | pending | - | - | - | - |
| qwen/qwen3-coder:free | pending | - | - | - | FREE |
```

**Update as each completes:**

```markdown
| Model | Status | Time | Issues | Quality | Cost |
|-------|--------|------|--------|---------|------|
| claude-embedded | success | 32s | 8 | 95% | FREE |
| x-ai/grok-code-fast-1 | success | 45s | 6 | 87% | $0.002 |
| qwen/qwen3-coder:free | timeout | - | - | - | - |
```

### Template B: Detailed Model Tracking (6+ models)

```markdown
## Model Execution Status

### Summary
- Total Requested: 8
- Completed: 0
- In Progress: 0
- Failed: 0
- Pending: 8

### Detailed Status

| # | Model | Provider | Status | Start | Duration | Issues | Quality | Cost | Error |
|---|-------|----------|--------|-------|----------|--------|---------|------|-------|
| 1 | claude-embedded | Anthropic | pending | - | - | - | - | FREE | - |
| 2 | x-ai/grok-code-fast-1 | X-ai | pending | - | - | - | - | - | - |
| 3 | qwen/qwen3-coder:free | Qwen | pending | - | - | - | - | FREE | - |
| 4 | google/gemini-3-pro | Google | pending | - | - | - | - | - | - |
| 5 | openai/gpt-5.1-codex | OpenAI | pending | - | - | - | - | - | - |
| 6 | mistralai/devstral | Mistral | pending | - | - | - | - | FREE | - |
| 7 | deepseek/deepseek-r1 | DeepSeek | pending | - | - | - | - | - | - |
| 8 | anthropic/claude-sonnet | Anthropic | pending | - | - | - | - | - | - |
```

### Template C: Session-Based Tracking File

Create this file at `$SESSION_DIR/tracking.md`:

```markdown
# Multi-Model Validation Tracking
Session: ${SESSION_ID}
Started: ${TIMESTAMP}

## Pre-Launch Verification
- [x] Session directory created: ${SESSION_DIR}
- [x] Tracking table initialized
- [x] Start time recorded: ${SESSION_START}
- [x] Model list: ${MODEL_LIST}

## Model Status

| Model | Status | Start | Duration | Issues | Quality |
|-------|--------|-------|----------|--------|---------|
| claude | pending | - | - | - | - |
| grok | pending | - | - | - | - |
| gemini | pending | - | - | - | - |

## Failures
(populated as failures occur)

## Consensus
(populated after all complete)
```

### Update Protocol

As each model completes, IMMEDIATELY update:

1. Status: `pending` -> `in_progress` -> `success`/`failed`/`timeout`
2. Duration: Calculate from start time
3. Issues: Number of issues found
4. Quality: Percentage if calculable
5. Error: If failed, brief error message

**DO NOT wait until all models finish.** Update as each completes.

---

## Per-Model Status Update Protocol

### IMMEDIATELY After Each Model Completes

Do NOT wait until all models finish. Update tracking AS EACH COMPLETES.

### Update Script

```bash
# Call this when each model completes
update_model_status() {
  local model="$1"
  local status="$2"
  local issues="${3:-0}"
  local quality="${4:-}"
  local error="${5:-}"

  local end_time=$(date +%s)
  local start_time="${MODEL_START_TIMES[$model]}"
  local duration=$((end_time - start_time))

  # Update arrays
  MODEL_END_TIMES["$model"]=$end_time
  MODEL_STATUS["$model"]="$status"

  # Log update to session tracking file
  echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) - Model: $model, Status: $status, Duration: ${duration}s" >> "$SESSION_DIR/execution.log"

  # Update tracking table (append to tracking.md)
  echo "| $model | $status | ${duration}s | $issues | ${quality:-N/A} | ${error:-} |" >> "$SESSION_DIR/tracking.md"

  # Track performance in global statistics
  if [[ "$status" == "success" ]]; then
    track_model_performance "$model" "success" "$duratio