Skip to main content

Debugging

Quick Diagnostics​

Run this complete health check:

#!/bin/bash
echo "=== Container Status ==="
docker compose ps

echo ""
echo "=== Redis Health ==="
docker exec agent-redis redis-cli ping

echo ""
echo "=== PostgreSQL Health ==="
docker exec agent-postgres psql -U agent -d agent -c "SELECT NOW();"

echo ""
echo "=== Stream Backlog ==="
docker exec agent-redis redis-cli XLEN events:incoming

echo ""
echo "=== Active Goals ==="
docker exec agent-redis redis-cli HLEN goals

echo ""
echo "=== Recent Errors (5 min) ==="
docker compose logs agent-core --since=5m 2>&1 | grep -i error | tail -10

echo ""
echo "=== Recent Scheduler Jobs ==="
docker compose logs agent-core --since=5m 2>&1 | grep "scheduler.job_complete" | tail -10

Debug Mode​

Enable verbose logging:

# Temporary (doesn't survive restart)
docker exec agent-core bash -c "LOG_LEVEL=DEBUG python -m src.main" &

# Permanent
LOG_LEVEL=DEBUG docker compose up -d agent-core

Debug mode outputs:

  • Full LLM prompt content
  • Memory retrieval results
  • Every skill call with full arguments
  • Redis read/write operations

Tracing a Single Request​

To trace what happens when you send a message:

# 1. Send the message
# 2. Watch logs in real-time
docker compose logs agent-core -f 2>&1 | grep -E "message_received|skill_call|response_sent|error"

Expected log sequence:

event_handler.message_received chat_id=987654321 text_len=45
event_handler.skill_call skill=web_search round=1
event_handler.skill_complete skill=web_search ms=1243
event_handler.response_sent tokens=387 rounds=2

Inspecting Context​

To see what the LLM receives, temporarily add logging to build_context():

# Temporary debug — revert after use
async def build_context(...):
    ...
    context = assembled_context
    logger.debug("build_context.assembled",
                 system_len=len(context[0].content),
                 history_len=len(context))
    return context

Redis Inspection​

# View all top-level keys
docker exec agent-redis redis-cli KEYS "*" | sort

# View goal state
docker exec agent-redis redis-cli HGETALL goals | python3 -c "
import sys, json
lines = sys.stdin.read().strip().split('\n')
for i in range(0, len(lines), 2):
    try:
        goal = json.loads(lines[i+1])
        print(f\"{goal['id'][:8]}... {goal['status']:10} {goal['objective'][:60]}\")
    except: pass
"

# View self-model
docker exec agent-redis redis-cli GET agent:self_model | python3 -m json.tool

# View epistemic state
docker exec agent-redis redis-cli GET agent:epistemic | python3 -m json.tool

# View behavioral rules queue
docker exec agent-redis redis-cli LLEN behavioral:pending
docker exec agent-redis redis-cli LRANGE behavioral:pending 0 -1

# View CPI
docker exec agent-redis redis-cli GET agent:cpi_current
docker exec agent-redis redis-cli GET agent:cpi_high

Database Inspection​

# Full database overview
docker exec agent-postgres psql -U agent -d agent << 'SQL'
SELECT
    t.tablename,
    c.reltuples::bigint AS row_count
FROM pg_tables t
JOIN pg_class c ON c.relname = t.tablename
WHERE t.schemaname = 'public'
ORDER BY c.reltuples DESC;
SQL

# Recent memory entries
docker exec agent-postgres psql -U agent -d agent -c "
  SELECT type, content[:80], created_at FROM memory_entries
  ORDER BY created_at DESC LIMIT 10;
"

# KG nodes
docker exec agent-postgres psql -U agent -d agent -c "
  SELECT entity_type, name, updated_at FROM knowledge_nodes
  ORDER BY updated_at DESC LIMIT 20;
"

# Recent skill calls
docker exec agent-postgres psql -U agent -d agent -c "
  SELECT skill_name, success, duration_ms, created_at FROM audit_log
  ORDER BY created_at DESC LIMIT 20;
"

Testing Skill Calls Directly​

Create a test script:

# /tmp/test_skill.py
import asyncio
import sys
sys.path.insert(0, '/app')

from src.skills.registry import SkillRegistry
from src.skills.builtin import register_builtin_skills
from src.memory.manager import MemoryManager
from src.config import settings

async def main():
    registry = SkillRegistry()
    memory = MemoryManager()
    register_builtin_skills(registry, memory, settings=settings)

    skill = registry.get('web_search')
    result = await skill.execute(query='Python 3.13 release notes', max_results=3)
    print("Output:", result.output[:500])
    print("Error:", result.error)

asyncio.run(main())
docker exec agent-core python3 /tmp/test_skill.py

Goal Engine Debugging​

# Watch goal ticks in real-time
docker compose logs agent-core -f | grep -E "goal_tick|goal_orchestrator|planning|task_executed"

# Check stuck goals
docker exec agent-redis redis-cli HGETALL goals | python3 -c "
import sys, json
lines = sys.stdin.read().strip().split('\n')
for i in range(0, len(lines)-1, 2):
    try:
        g = json.loads(lines[i+1])
        if g.get('status') in ['active', 'planning']:
            age = '???'
            print(f\"{g['id'][:8]} | {g['status']:10} | {g['objective'][:60]}\")
    except: pass
"

Memory Debugging​

# Search memories by content
docker exec agent-postgres psql -U agent -d agent -c "
  SELECT type, content, created_at FROM memory_entries
  WHERE content ILIKE '%bitcoin%'
  ORDER BY created_at DESC LIMIT 10;
"

# Check knowledge graph
docker exec agent-postgres psql -U agent -d agent -c "
  SELECT n.entity_type, n.name, COUNT(r.id) as relation_count
  FROM knowledge_nodes n
  LEFT JOIN knowledge_relations r ON r.from_node = n.id OR r.to_node = n.id
  GROUP BY n.id, n.entity_type, n.name
  ORDER BY relation_count DESC
  LIMIT 20;
"

Container Resource Usage​

# Real-time stats
docker stats --no-stream

# Check shared memory (for Chromium)
docker exec agent-core df -h /dev/shm

Network Debugging​

# Check container can reach internet
docker exec agent-core curl -s https://api.coinbase.com/v2/prices/BTC-USD/spot

# Check internal network
docker exec agent-core curl -s http://agent-redis:6379  # Should get Redis response
docker exec agent-core curl -s http://agent-postgres:5432  # Should fail (non-HTTP)
docker exec agent-core curl -s http://agent-core:8080/health  # Should get 200

Reporting Issues​

When reporting an issue, include:

# Collect diagnostics
echo "=== Version ===" && docker inspect agent-core | python3 -c "
import sys,json
c=json.load(sys.stdin)[0]
print('Image:', c['Config']['Image'])
print('Created:', c['Created'])
"
echo "=== Last 50 Lines ===" && docker compose logs agent-core --tail=50
echo "=== Container Stats ===" && docker stats --no-stream --format "{{.Name}}: {{.CPUPerc}} CPU, {{.MemUsage}} RAM"