Advanced Error Reporting

Detailed Test Failure Analysis & Debugging

MCP Aegis provides comprehensive error reporting with detailed validation analysis, precise error locations, and actionable suggestions to help you quickly identify and fix test issues.

🔍 Enhanced Validation System

When tests fail, MCP Aegis's enhanced validation system provides:

✅ Precise Error Locations - Exact path to the failing validation (e.g., response.result.tools[0].inputSchema.required)
✅ Error Categorization - Structured error types for better understanding
✅ Actionable Suggestions - Specific recommendations for fixing each error
✅ Value Previews - Shows expected vs actual values with intelligent truncation
✅ Error Grouping - Similar errors are grouped with counts for clarity
✅ Top Recommendations - Prioritized suggestions based on error frequency

📊 Error Types & Categories

MCP Aegis categorizes validation errors into specific types to help you understand exactly what went wrong:

Structural Errors

Missing Field Errors

When expected fields are missing from the server response.

yaml

# ❌ Test expects 'name' field but server doesn't provide it
expect:
  response:
    result:
      tool:
        name: "read_file"  # Expected but missing
        description: "Reads files"

Extra Field Errors

When server returns fields not specified in expectations (only for exact matching).

yaml

# ❌ Server returns extra 'version' field not in expectations
# Server Response: {"name": "read_file", "version": "1.0"}
expect:
  response:
    result:
      tool:
        name: "read_file"  # Missing 'version' in expectations

Validation Errors

Pattern Failed Errors

When pattern matching validation fails (regex, length, type, etc.).

yaml

# ❌ Pattern doesn't match actual value
expect:
  response:
    result:
      message: "match:startsWith:Success"  # But actual: "Error occurred"

Value Mismatch Errors

When exact values don't match expected values.

Type Mismatch Errors

When data types don't match expected types.

Length Mismatch Errors

When array/string lengths don't match expectations.

📋 Example Error Output

Here's what a typical enhanced error report looks like:

bash

● should have all expected tools with proper structure ... ✗ FAIL
    At response.result.tools[0].inputSchema.required: Unexpected field 'required' (5 additional validation errors found)
    
    🔍 Detailed Validation Analysis:
    📊 Found 6 unexpected field(s).

    ➕ EXTRA FIELD
       📍 Path: response.result.tools[0].inputSchema.required
       💬 Unexpected field 'required'
       💡 Suggestion: Remove 'required' from server response or add it to expected response with value: ["url"]
       
    ➕ EXTRA FIELD
       📍 Path: response.result.tools[1].inputSchema.required
       💬 Unexpected field 'required'
       💡 Suggestion: Remove 'required' from server response or add it to expected response with value: ["responseData","analysis"]
       
    🎯 Top Recommendations:
    1. Remove 'required' from server response or add it to expected response with value: ["url"] (6 similar issues found)

🎯 Error Output Sections

Summary Line

The first line provides a quick overview of the primary error and indicates additional errors:

bash

At response.result.tools[0].inputSchema.required: Unexpected field 'required' (5 additional validation errors found)

Detailed Analysis

The detailed section shows error statistics and categorization:

bash

🔍 Detailed Validation Analysis:
📊 Found 6 unexpected field(s).

Individual Error Details

Each error includes:

Error Icon - Visual indicator of error type (➕ ➖ ❌ ⚠️ 🔢 📝)
Path - Exact location of the error in the response
Message - Clear description of what went wrong
Suggestion - Specific actionable recommendation
Values - Expected vs actual values when relevant

Top Recommendations

Prioritized suggestions based on error frequency and impact:

bash

🎯 Top Recommendations:
1. Remove 'required' from server response or add it to expected response with value: ["url"] (6 similar issues found)
2. Update expected response to match server structure (3 similar issues found)

🎨 Error Type Icons

Icon	Error Type	Description
➖	MISSING FIELD	Expected field is missing from server response
➕	EXTRA FIELD	Server response has unexpected field
❌	PATTERN FAILED	Pattern matching validation failed
⚠️	VALUE MISMATCH	Expected and actual values don't match
🔢	TYPE MISMATCH	Data type doesn't match expectation
📝	LENGTH MISMATCH	Array or string length doesn't match

🛠️ Using Error Reports for Debugging

Quick Fixes

Most error reports include specific suggestions you can apply immediately:

✅ For Extra Field Errors:

Add the field to your expected response
Use match:partial: for flexible validation
Modify server to not return the field

🔵 For Missing Field Errors:

Update server to include the field
Remove the field from expected response
Make the field optional in your schema

🟣 For Pattern Failed Errors:

Check the pattern syntax
Verify actual server response format
Use --debug to see raw responses

Debugging Workflow

Run test - Execute your test to get error report
Review summary - Look at the primary error and count
Check paths - Identify exactly where errors occur
Apply suggestions - Use the provided recommendations
Re-run test - Verify fixes worked

🔧 Advanced Debugging Techniques

Using Debug Mode

Use --debug to see raw MCP communication:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --debug

Using Verbose Mode

Use --verbose for detailed test hierarchy:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --verbose

Combining Debug Modes

Combine modes for comprehensive debugging information:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --verbose --debug --timing

Error Reporting Options

Use specialized error reporting options for focused debugging:

Show Only Failed Tests

Focus on failures by hiding passing tests:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --errors-only

Syntax Error Analysis

Focus on pattern syntax issues and get correction suggestions:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --syntax-only

Minimal Error Output

Get basic error messages without detailed analysis:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --no-analysis

Group Similar Errors

Group identical errors together to reduce repetition:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --group-errors

Limit Error Count

Limit the number of validation errors shown per test:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --max-errors 3

Combining Error Options

Combine multiple error reporting options for focused debugging:

bash

aegis "tests/*.test.mcp.yml" --config "config.json" --errors-only --group-errors --max-errors 2

🎯 Partial Matching for Extra Fields

One of the most common error scenarios is when servers return more fields than expected. Instead of updating all test expectations, use partial matching:

yaml

# ❌ BEFORE: Fails when server adds extra fields
expect:
  response:
    result:
      tools:
        match:arrayElements:
          name: match:type:string
          description: match:type:string
          inputSchema:
            type: object
            properties: match:type:object
            # ❌ Server also returns 'required' field - causes failure

# ✅ AFTER: Uses partial matching to ignore extra fields
expect:
  response:
    result:
      tools:
        match:arrayElements:
          name: match:type:string
          description: match:type:string
          inputSchema:
            match:partial:  # 🎯 Only validates specified fields
              type: object
              properties: match:type:object

🛡️ Error Prevention Best Practices

Start simple - Begin with basic exact matching, then add patterns
Use partial matching - When you only care about specific fields
Test incrementally - Add one validation at a time
Validate server responses - Use --debug to see actual output
Group related tests - Separate structure validation from content validation
Use meaningful test names - Help identify intent when debugging
Check YAML syntax - Use YAML linters to catch formatting issues

❗ Common Error Patterns & Solutions

Schema Evolution

Problem: Server adds new fields, breaking existing tests

Solution: Use match:partial: for forward compatibility

Array vs Object Confusion

Problem: Expecting array but getting single object

Solution: Check actual server response structure with --debug

Duplicate YAML Keys

Problem: YAML overwrites duplicate keys silently

Solution: Use YAML linters and structure patterns correctly

yaml

# ❌ WRONG: Duplicate keys (second overwrites first)
result:
  tools: "match:arrayLength:1"
  tools: ["read_file"]  # Overwrites the above!

# ✅ CORRECT: Separate tests or proper structure
result:
  tools: "match:arrayLength:1"
# Test arrayLength in one test, specific tools in another

🤝 Getting Help

If error reports don't provide enough information:

Run with --debug --verbose for full context
Check your YAML syntax with a linter
Verify your server is responding as expected
Compare with working examples in the MCP Aegis repository
Create minimal reproduction cases for complex issues

💡 Pro Tip: The enhanced error reporting system is designed to make debugging as straightforward as possible. Most issues can be resolved by following the specific suggestions in the error output.