📝 YAML Testing Guide

Declarative Model Context Protocol Testing

Human-Readable Testing with YAML

Write declarative tests using simple YAML files with 50+ powerful pattern matching types. Perfect for manual test creation, CI/CD integration, and comprehensive MCP server validation.

50+ PatternsDeclarativeNo Coding Required

🚀 Quick Start: Your First YAML Test

Learn by example! Create your first YAML test in under 2 minutes with this step-by-step guide.

📋 Prerequisites

✅ MCP Aegis installed

✅ Working MCP server

✅ Configuration file (aegis.config.json)

Need help? See Quick Start Guide for installation.

📝 Step 1: Create Your First Test File

Create first.test.mcp.yml with this simple test:

yaml

description: "My first MCP YAML test"
tests:
  - it: "should list available tools"
    request:
      jsonrpc: "2.0"
      id: "test-1"
      method: "tools/list"
      params: {}
    expect:
      response:
        jsonrpc: "2.0"
        id: "test-1"
        result:
          tools: "match:arrayLength:1"  # Expect exactly 1 tool
      stderr: "toBeEmpty"

🏃 Step 2: Run Your Test

bash

aegis first.test.mcp.yml --config aegis.config.json

✅ Expected output: 1 passed

🔍 Step 3: Add Tool Execution Test

Extend your test to actually call a tool:

yaml

- it: "should execute read_file tool successfully"
    request:
      jsonrpc: "2.0"
      id: "test-2"
      method: "tools/call"
      params:
        name: "read_file"
        arguments:
          path: "../shared-test-data/hello.txt"  # Use existing test file
    expect:
      response:
        jsonrpc: "2.0"
        id: "test-2"
        result:
          content:
            - type: "text"
              text: "match:type:string"  # Expect string content
          isError: false
      stderr: "toBeEmpty"

🎯 Step 4: Use Pattern Matching

Make your tests more flexible with pattern matching:

yaml

- it: "should validate tool structure"
    request:
      jsonrpc: "2.0"
      id: "test-3"
      method: "tools/list"
      params: {}
    expect:
      response:
        result:
          tools:
            match:arrayElements:
              name: "match:type:string"
              description: "match:regex:.{10,}"  # At least 10 chars
              inputSchema: "match:type:object"

💡 This validates that ALL tools have proper structure!

CLI Options

MCP Aegis provides several CLI options for debugging and different output formats:

bash

# Basic test execution
aegis "tests/*.yml" --config config.json

# Verbose output shows test hierarchy and individual results
aegis "tests/*.yml" --config config.json --verbose

# Debug mode shows detailed MCP communication (JSON-RPC messages)
aegis "tests/*.yml" --config config.json --debug

# Timing information for performance analysis
aegis "tests/*.yml" --config config.json --timing

# JSON output for CI/automation systems
aegis "tests/*.yml" --config config.json --json

# Quiet mode suppresses non-essential output
aegis "tests/*.yml" --config config.json --quiet

# Combine multiple debugging options
aegis "tests/*.yml" --config config.json --verbose --debug --timing

Option	Short	Description
`--config`	`-c`	Path to configuration file (default: `./aegis.config.json`)
`--verbose`	`-v`	Display individual test results with test suite hierarchy
`--debug`	`-d`	Enable debug mode with detailed MCP communication logging
`--timing`	`-t`	Show timing information for tests and operations
`--json`	`-j`	Output results in JSON format for CI/automation
`--quiet`	`-q`	Suppress non-essential output (opposite of verbose)

Advanced & Error-Focused Flags

Use these flags to focus on failures, control analysis depth, and manage output volume:

Option	Description	Notes / Interactions
`--errors-only`	Show only failing tests (hides passes)	Incompatible with `--verbose`
`--group-errors`	Group identical failures to reduce repetition	Great with `--errors-only` for large suites
`--syntax-only`	Only perform pattern/YAML syntax analysis (no execution)	Mutually exclusive with `--no-analysis`
`--no-analysis`	Disable deep pattern diagnostics (faster, minimal errors)	Mutually exclusive with `--syntax-only`
`--max-errors <n>`	Limit number of reported errors (default 5)	Use higher during triage: `--max-errors 20`
`--concise`	Compact output (suppresses extra spacing & headers)	Pairs well with `--errors-only`
`--json`	Structured machine-readable output	Disables verbose output for cleanliness

JSON Output & Grouping

The --json flag emits a structured object suitable for CI parsing. When combined with --group-errors, the JSON includes an errorGroups array that consolidates repeated failures:

json

{
  "summary": {"passed": 18, "failed": 3, "durationMs": 742},
  "tests": [
    {"id": "tools-1", "status": "passed"},
    {"id": "exec-1", "status": "failed", "errorGroupId": 1}
  ],
  "errorGroups": [
    {
      "id": 1,
      "count": 3,
      "message": "Pattern mismatch: tools[0].name",
      "firstTestId": "exec-1"
    }
  ]
}

Tip: Use --errors-only --group-errors --json --max-errors 50 for high-signal CI logs on large suites.

Output Examples

Verbose Output (--verbose):

bash

📋 Test Results Hierarchy:

📁 Calculator Tests (15ms)
   tests/calculator.test.mcp.yml

  ✓ should perform addition (2ms)
  ✓ should handle division
  ✗ should validate input (1ms)

Debug Output (--debug):

bash

📡 [MCP SEND] → tools/call
    {
      "jsonrpc": "2.0",
      "id": "calc-1",
      "method": "tools/call",
      "params": {
        "name": "calculator",
        "arguments": { "a": 15, "b": 27 }
      }
    }
📡 [MCP RECV] ← response
    {
      "jsonrpc": "2.0",
      "id": "calc-1", 
      "result": {
        "content": [{"type": "text", "text": "Result: 42"}]
      }
    }

📁 Test File Structure

YAML test files follow a consistent, predictable structure that makes them easy to read and maintain.

📋 Basic Structure

yaml

description: "Human-readable test suite description"
tests:
  - it: "Individual test case description"
    request:
      jsonrpc: "2.0"
      id: "unique-test-identifier"
      method: "tools/list|tools/call|initialize"
      params:
        # Method-specific parameters
    expect:
      response:
        jsonrpc: "2.0"
        id: "unique-test-identifier"
        result:
          # Expected response structure
      stderr: "toBeEmpty"  # Optional stderr validation

✅ Required Fields

description: Test suite description
tests: Array of test cases
it: What the test validates
request: JSON-RPC request to send
expect: Expected response structure

🔧 Optional Fields

stderr: Expected stderr output
timeout: Custom timeout (ms)
skip: Skip test (debugging)
only: Run only this test

💡 Pro Tips

• Use unique IDs for each test to avoid conflicts
• Include stderr: "toBeEmpty" for clean tests
• Group related tests in the same file
• Use descriptive test names that explain the expected behavior

📝 Naming Conventions

File Names

text

✅ Good:
filesystem.test.mcp.yml
multi-tool.test.mcp.yml
api-testing.test.mcp.yml

❌ Avoid:
test.yml
mytest.yaml
server_test.yml

Test IDs

text

✅ Good:
"list-1", "exec-1", "error-1"
"calc-add", "calc-div"
"fs-read", "fs-write"

❌ Avoid:
"1", "test", "t1"
"abc", "xyz"

🎯 Pattern Matching Power

YAML testing becomes powerful through pattern matching - flexible validation that adapts to dynamic server responses.

🚀 Why Pattern Matching?

❌ Brittle Testing

• Exact text matching breaks on version changes

• Hard-coded array lengths fail with updates

• Server timestamps cause test flakiness

✅ Flexible Validation

• "match:contains:success" - partial text

• "match:arrayLength:3" - exact counts

• "match:regex:\\d4-\\d2-\\d2" - date formats

Essential Patterns

Master these 6 patterns to handle 90% of your testing needs:

🔢 Array Validation

yaml

# Exact length
tools: "match:arrayLength:3"

# All elements have structure
tools:
  match:arrayElements:
    name: "match:type:string"
    description: "match:regex:.{10,}"

# Contains specific item
toolNames: "match:arrayContains:read_file"

📝 String Validation

yaml

# Contains text
message: "match:contains:success"

# Starts/ends with
filename: "match:startsWith:data_"
extension: "match:endsWith:.json"

# Pattern matching
version: "match:regex:v\d+\.\d+\.\d+"

🎯 Field Extraction

yaml

# Extract all tool names
result:
  match:extractField: "tools.*.name"
  value:
    - "calculator"
    - "text_processor"

# Extract with validation
match:extractField: "content.*.type"
value: "match:arrayContains:text"

📋 Partial Matching

yaml

# Only validate specific fields
result:
  match:partial:
    tools:
      - name: "read_file"
        description: "match:contains:file"
    # Ignores other response fields

💡 Pro Tip: See the complete Pattern Matching Reference for all 50+ patterns including numeric, date, cross-field, and negation patterns.

📋 Common Test Patterns

Ready-to-use test patterns based on real-world MCP server implementations from our examples directory.

🔍 1. Tool Discovery

Validate that your server exposes tools correctly with proper schemas.

yaml

description: "Tool discovery validation"
tests:
  - it: "should list available tools with proper structure"
    request:
      jsonrpc: "2.0"
      id: "list-1"
      method: "tools/list"
      params: {}
    expect:
      response:
        jsonrpc: "2.0"
        id: "list-1"
        result:
          tools:
            match:arrayElements:
              name: "match:regex:^[a-z][a-z0-9_]*quot;    # snake_case validation
              description: "match:regex:.{10,}"         # Min 10 chars
              inputSchema:
                type: "object"
                properties: "match:type:object"
                required: "match:type:array"
      stderr: "toBeEmpty"

✨ Real Example: From examples/filesystem-server/filesystem.test.mcp.yml

⚡ 2. Tool Execution

Test actual tool functionality with flexible response validation.

yaml

- it: "should execute read_file tool successfully"
    request:
      jsonrpc: "2.0"
      id: "exec-1"
      method: "tools/call"
      params:
        name: "read_file"
        arguments:
          path: "../shared-test-data/hello.txt"
    expect:
      response:
        jsonrpc: "2.0"
        id: "exec-1"
        result:
          content:
            - type: "text"
              text: "Hello, MCP Aegis!"      # Exact match for demo
          isError: false
      stderr: "toBeEmpty"

🎯 Flexible Version: Use "match:contains:Hello" for partial text matching

❌ 3. Error Handling

Ensure your server handles invalid inputs gracefully.

yaml

- it: "should handle non-existent file gracefully"
    request:
      jsonrpc: "2.0"
      id: "error-1"
      method: "tools/call"
      params:
        name: "read_file"
        arguments:
          path: "nonexistent.txt"
    expect:
      response:
        jsonrpc: "2.0"
        id: "error-1"
        result:
          content:
            - type: "text"
              text: "match:contains:not found"   # Flexible error message
          isError: true
      stderr: "toBeEmpty"

🛠️ 4. Multi-Tool Validation

Test complex servers with multiple tools efficiently.

yaml

- it: "should have exactly 4 tools with proper naming"
    request:
      jsonrpc: "2.0"
      id: "multi-1"
      method: "tools/list"
      params: {}
    expect:
      response:
        result:
          tools: "match:arrayLength:4"          # Exact count
        
  - it: "should extract all tool names correctly"
    request:
      jsonrpc: "2.0"
      id: "multi-2"
      method: "tools/list"
      params: {}
    expect:
      response:
        result:
          match:extractField: "tools.*.name"    # Extract all names
          value:
            - "calculator"
            - "text_processor"
            - "data_validator"
            - "file_manager"

🚀 Real Example: From examples/multi-tool-server/multi-tool.test.mcp.yml

🎯 5. Advanced Response Validation

Use partial matching and complex patterns for sophisticated validation.

yaml

- it: "should validate calculator response structure"
    request:
      jsonrpc: "2.0"
      id: "calc-1"
      method: "tools/call"
      params:
        name: "calculator"
        arguments:
          operation: "add"
          a: 15
          b: 27
    expect:
      response:
        result:
          match:partial:                        # Only validate these fields
            content:
              - type: "text"
                text: "match:regex:Result: \d+"  # "Result: 42"
            isError: false
      stderr: "toBeEmpty"

📚

More Examples

See complete test suites in action

🎯

Pattern Guide

Master all 50+ pattern types

🤖

AI Support

Get AI assistance for test generation

🎯 Best Practices

Follow these proven practices to write maintainable, reliable YAML tests.

✅ Do This

✓
Use descriptive test names:
"should list 3 available tools"
✓
Include initialization tests:
Test MCP handshake before tools
✓
Test both success and error scenarios:
Validate error handling paths
✓
Use pattern matching:
Avoid brittle exact matches

❌ Avoid This

✗
Hard-coded exact text:
"Error: File not found at /tmp/xyz"
✗
Non-unique test IDs:
Multiple tests with id: "test-1"
✗
Missing error validation:
Not testing invalid inputs
✗
Duplicate YAML keys:
YAML silently overwrites duplicates

📂 Test Organization

text

project/
├── test/mcp/yaml/
│   ├── basic-functionality.test.mcp.yml    # Core features
│   ├── error-handling.test.mcp.yml         # Error scenarios  
│   ├── performance.test.mcp.yml            # Performance tests
│   └── edge-cases.test.mcp.yml             # Boundary conditions
└── aegis.config.json                   # Server configuration

🎯 Pattern Selection Guide

Use Case	Pattern	Example
Tool count validation	`arrayLength`	`"match:arrayLength:3"`
Success messages	`contains`	`"match:contains:success"`
Version strings	`regex`	`"match:regex:v\\d+\\.\\d+"`
Tool name extraction	`extractField`	`"tools.*.name"`
Schema validation	`arrayElements`	Validate all tool structures

🛠️ Troubleshooting Guide

Quick solutions to common YAML testing issues and debugging techniques.

⚠️ YAML Structure Issues

Common Mistakes

❌ Duplicate Keys

yaml

result:
  tools: "match:arrayLength:1"
  tools: ["read_file"]  # OVERWRITES previous!

✅ Solution

yaml

# Separate tests for different validations
- it: "should have correct array length"
  expect:
    response:
      result:
        tools: "match:arrayLength:1"

Critical Guidelines

• Consistent indentation: Use 2 spaces, no tabs
• Quote patterns: Always quote match:* patterns
• Escape regex: Use \\d+ not \d+
• Unique keys: No duplicate keys in same object

🔍 Pattern Debugging

Problem: Pattern Mismatch

Error: Pattern did not match: expected "match:regex:..." but got "..."

🔧 Debug Steps

1. Use --debug flag:
aegis test.yml --debug
2. Check actual response:
Look at the JSON-RPC output
3. Validate YAML syntax:
Use online YAML validators
4. Test incrementally:
Start simple, add complexity

🎯 Quick Fixes

Regex escaping:
"match:regex:\\\\d+" ← Double escape

String contains:
Use "match:contains:text" for partial matches

Array validation:
Check array vs object mismatch

🔌 Server Connection Issues

Problem: Connection Timeout

Error: Server failed to start or connection timeout

🔧 Solutions

1. Check configuration:
Verify command and args
2. Increase timeout:
Add "startupTimeout": 10000
3. Add ready pattern:
Use "readyPattern": "Server ready"
4. Check server logs:
Use --debug to see stderr

📝 Config Example

json

{
  "name": "My Server",
  "command": "node",
  "args": ["server.js"],
  "startupTimeout": 10000,
  "readyPattern": "Server listening"
}

🚨 Common Anti-Patterns

Anti-Pattern	Problem	Solution
Exact text matching	Breaks on updates	Use `match:contains`
Missing stderr validation	Ignores errors	Add `stderr: "toBeEmpty"`
Non-unique test IDs	Conflicts	Use descriptive unique IDs
Complex nested patterns	Hard to debug	Split into separate tests

MCP

MCP

📝 YAML Testing Guide

Human-Readable Testing with YAML

🚀 Quick Start: Your First YAML Test

📋 Prerequisites

📝 Step 1: Create Your First Test File

🏃 Step 2: Run Your Test

🔍 Step 3: Add Tool Execution Test

🎯 Step 4: Use Pattern Matching

CLI Options

Advanced & Error-Focused Flags

JSON Output & Grouping

Output Examples

📁 Test File Structure

📋 Basic Structure

✅ Required Fields

🔧 Optional Fields

💡 Pro Tips

📝 Naming Conventions

File Names

Test IDs

🎯 Pattern Matching Power

🚀 Why Pattern Matching?

❌ Brittle Testing

✅ Flexible Validation

Essential Patterns

🔢 Array Validation

📝 String Validation

🎯 Field Extraction

📋 Partial Matching

📋 Common Test Patterns

🔍 1. Tool Discovery

⚡ 2. Tool Execution

❌ 3. Error Handling

🛠️ 4. Multi-Tool Validation

🎯 5. Advanced Response Validation

More Examples

Pattern Guide

AI Support

🎯 Best Practices

✅ Do This

❌ Avoid This

📂 Test Organization

🎯 Pattern Selection Guide

🛠️ Troubleshooting Guide

⚠️ YAML Structure Issues

Common Mistakes

Critical Guidelines

🔍 Pattern Debugging

Problem: Pattern Mismatch

🔧 Debug Steps

🎯 Quick Fixes

🔌 Server Connection Issues

Problem: Connection Timeout

🔧 Solutions

📝 Config Example

🚨 Common Anti-Patterns

🎯 Need More Help?