Validation Framework Tutorial

This notebook demonstrates the new rich parameter validation framework in epimodels.

Overview

The validation framework provides:

  1. Declarative Parameter Specifications: Define parameters with types, bounds, constraints

  2. Constraint Language: Express relationships between parameters

  3. Severity Levels: Error vs warning constraints

  4. Backward Compatibility: Existing models work unchanged

[1]:

from epimodels import BaseModel, ValidationError
from epimodels.validation import (
    ParameterSpec,
    VariableSpec,
    ModelConstraint,
    DomainType,
    evaluate_constraint,
)
from collections import OrderedDict
import warnings

1. Parameter Specifications

Define parameters with rich metadata including bounds, constraints, units, and documentation.

[2]:

# Create a parameter specification
beta_spec = ParameterSpec(
    name="beta",
    symbol=r"$\beta$",
    description="Transmission rate (contact rate × probability of transmission)",
    domain_type=DomainType.CONTINUOUS,
    bounds=(0, None),  # Must be positive
    dtype=float,
    required=True,
    constraints=["value > 0"],
    units="1/day",
    typical_range=(0.1, 1.0)
)

print(f"Parameter: {beta_spec.name}")
print(f"Symbol: {beta_spec.symbol}")
print(f"Description: {beta_spec.description}")
print(f"Bounds: {beta_spec.bounds}")
print(f"Constraints: {beta_spec.constraints}")
print(f"Units: {beta_spec.units}")

Parameter: beta
Symbol: $\beta$
Description: Transmission rate (contact rate × probability of transmission)
Bounds: (0, None)
Constraints: ['value > 0']
Units: 1/day

2. Creating a Model with Rich Validation

[3]:

class SIRWithValidation(BaseModel):
    """SIR model with rich parameter validation."""

    def __init__(self):
        super().__init__()
        self.model_type = "SIR"

        # Define parameters with rich specifications
        self.define_parameter(ParameterSpec(
            name="beta",
            symbol=r"$\beta$",
            description="Transmission rate",
            bounds=(0, None),
            constraints=["value > 0"],
            units="1/day",
            typical_range=(0.1, 1.0)
        ))

        self.define_parameter(ParameterSpec(
            name="gamma",
            symbol=r"$\gamma$",
            description="Recovery rate (1 / average infectious period)",
            bounds=(0, None),
            constraints=["value > 0"],
            units="1/day",
            typical_range=(0.05, 0.5)
        ))

        # Define state variables
        self.define_variable(VariableSpec(
            name="S",
            symbol="S",
            description="Susceptible individuals",
            non_negative=True,
            units="individuals"
        ))

        self.define_variable(VariableSpec(
            name="I",
            symbol="I",
            description="Infectious individuals",
            non_negative=True,
            units="individuals"
        ))

        self.define_variable(VariableSpec(
            name="R",
            symbol="R",
            description="Recovered/Removed individuals",
            non_negative=True,
            units="individuals"
        ))

        # Add model-level constraints
        self.add_constraint(ModelConstraint(
            expression="beta / gamma > 1",
            description="R0 > 1 required for epidemic spread",
            severity="warning",
            name="R0_epidemic"
        ))

# Create model instance
model = SIRWithValidation()
print(f"Model type: {model.model_type}")
print(f"Parameters: {list(model.parameter_specs.keys())}")
print(f"Variables: {list(model.variable_specs.keys())}")
print(f"Constraints: {len(model.model_constraints)}")

Model type: SIR
Parameters: ['beta', 'gamma']
Variables: ['S', 'I', 'R']
Constraints: 1

3. Parameter Validation

[4]:

# Test 1: Valid parameters
print("Test 1: Valid parameters")
try:
    model.validate_parameters({'beta': 0.3, 'gamma': 0.1})
    print("  ✓ Validation passed")
except ValidationError as e:
    print(f"  ✗ Validation failed: {e}")

Test 1: Valid parameters
  ✓ Validation passed

[5]:

# Test 2: Missing required parameter
print("Test 2: Missing required parameter")
try:
    model.validate_parameters({'beta': 0.3})
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {type(e).__name__}")
    print(f"    Message: {str(e)[:60]}...")

Test 2: Missing required parameter
  ✓ Caught error: ValidationError
    Message: Missing required parameter: gamma...

/home/fccoelho/Documentos/Projects_software/epimodels/epimodels/__init__.py:123: UserWarning: Constraint violated: R0 > 1 required for epidemic spread (Failed to evaluate expression: Unknown variable: gamma)
  warnings.warn(warning_msg, UserWarning)

[6]:

# Test 3: Parameter out of bounds
print("Test 3: Parameter out of bounds (negative beta)")
try:
    model.validate_parameters({'beta': -0.3, 'gamma': 0.1})
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {type(e).__name__}")
    print(f"    Message: {str(e)[:60]}...")

Test 3: Parameter out of bounds (negative beta)
  ✓ Caught error: ValidationError
    Message: Parameter 'beta' value -0.3 is below minimum bound 0
Paramet...

/home/fccoelho/Documentos/Projects_software/epimodels/epimodels/__init__.py:123: UserWarning: Constraint violated: R0 > 1 required for epidemic spread
  warnings.warn(warning_msg, UserWarning)

[7]:

# Test 4: Warning constraint (R0 < 1)
print("Test 4: Warning constraint (R0 < 1)")
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    try:
        model.validate_parameters({'beta': 0.1, 'gamma': 0.3})
        if w:
            print(f"  ✓ Warning raised: {str(w[0].message)[:60]}...")
        else:
            print("  ✗ No warning raised")
    except ValidationError as e:
        print(f"  ✗ Validation failed: {e}")

Test 4: Warning constraint (R0 < 1)
  ✓ Warning raised: Constraint violated: R0 > 1 required for epidemic spread...

[8]:

# Test 5: Constraint violation (gamma = 0)
print("Test 5: Constraint violation (gamma = 0)")
try:
    model.validate_parameters({'beta': 0.3, 'gamma': 0})
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {type(e).__name__}")
    print(f"    Message: {str(e)[:60]}...")

Test 5: Constraint violation (gamma = 0)
  ✓ Caught error: ValidationError
    Message: Parameter 'gamma' value 0.0 violates constraint: value > 0...

/home/fccoelho/Documentos/Projects_software/epimodels/epimodels/__init__.py:123: UserWarning: Constraint violated: R0 > 1 required for epidemic spread (Failed to evaluate expression: float division by zero)
  warnings.warn(warning_msg, UserWarning)

4. Initial Condition Validation

[9]:

# Test 6: Valid initial conditions
print("Test 6: Valid initial conditions")
try:
    model.validate_initial_conditions([990, 10, 0], totpop=1000)
    print("  ✓ Initial conditions valid")
except ValidationError as e:
    print(f"  ✗ Validation failed: {e}")

Test 6: Valid initial conditions
  ✓ Initial conditions valid

[10]:

# Test 7: Negative initial condition
print("Test 7: Negative initial condition")
try:
    model.validate_initial_conditions([990, -10, 0], totpop=1000)
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {type(e).__name__}")

Test 7: Negative initial condition
  ✓ Caught error: ValidationError

[11]:

# Test 8: Initial conditions exceed population
print("Test 8: Initial conditions exceed population")
try:
    model.validate_initial_conditions([1000, 100, 0], totpop=1000)
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {type(e).__name__}")

Test 8: Initial conditions exceed population
  ✓ Caught error: ValidationError

5. Constraint Language

The constraint evaluator supports various operators and expressions.

[12]:

# Test various constraint expressions
constraints = [
    ("beta > gamma", {'beta': 0.5, 'gamma': 0.1}),
    ("beta / gamma > 1", {'beta': 0.3, 'gamma': 0.1}),
    ("p + q <= 1", {'p': 0.6, 'q': 0.3}),
    ("rate**2 > threshold", {'rate': 2, 'threshold': 3}),
    ("x > 0 and y > 0", {'x': 1, 'y': 1}),
    ("(alpha + beta) / gamma > 2", {'alpha': 0.2, 'beta': 0.3, 'gamma': 0.2}),
]

print("Testing constraint expressions:")
print("-" * 60)
for expr, params in constraints:
    satisfied, msg = evaluate_constraint(expr, params)
    status = "✓" if satisfied else "✗"
    print(f"{status} '{expr}' with {params}: {satisfied}")

Testing constraint expressions:
------------------------------------------------------------
✓ 'beta > gamma' with {'beta': 0.5, 'gamma': 0.1}: True
✓ 'beta / gamma > 1' with {'beta': 0.3, 'gamma': 0.1}: True
✓ 'p + q <= 1' with {'p': 0.6, 'q': 0.3}: True
✓ 'rate**2 > threshold' with {'rate': 2, 'threshold': 3}: True
✓ 'x > 0 and y > 0' with {'x': 1, 'y': 1}: True
✓ '(alpha + beta) / gamma > 2' with {'alpha': 0.2, 'beta': 0.3, 'gamma': 0.2}: True

6. More Complex Model Example

[13]:

class SEQIAHRWithValidation(BaseModel):
    """COVID-19 model with quarantine and hospitalization."""

    def __init__(self):
        super().__init__()
        self.model_type = "SEQIAHR"

        # Transmission parameters
        self.define_parameter(ParameterSpec(
            name="beta",
            symbol=r"$\beta$",
            description="Transmission rate",
            bounds=(0, None),
            constraints=["value > 0"],
            units="1/day"
        ))

        # Probabilities (must be between 0 and 1)
        self.define_parameter(ParameterSpec(
            name="p",
            symbol="p",
            description="Proportion asymptomatic",
            bounds=(0, 1),
            constraints=["value >= 0", "value <= 1"],
            units="dimensionless"
        ))

        self.define_parameter(ParameterSpec(
            name="chi",
            symbol=r"$\chi$",
            description="Quarantine effectiveness",
            bounds=(0, 1),
            constraints=["value >= 0", "value <= 1"],
            units="dimensionless"
        ))

        # Rates
        self.define_parameter(ParameterSpec(
            name="gamma",
            symbol=r"$\gamma$",
            description="Recovery rate",
            bounds=(0, None),
            constraints=["value > 0"],
            units="1/day"
        ))

        self.define_parameter(ParameterSpec(
            name="mu",
            symbol=r"$\mu$",
            description="Mortality rate",
            bounds=(0, None),
            constraints=["value >= 0"],
            units="1/day"
        ))

        # Add constraint: mortality + recovery should be reasonable
        self.add_constraint(ModelConstraint(
            expression="mu + gamma < 1",
            description="Combined exit rate should be < 1/day for stability",
            severity="warning"
        ))

        # Add constraint: quarantine effectiveness
        self.add_constraint(ModelConstraint(
            expression="chi >= 0.5",
            description="Quarantine should be at least 50% effective",
            severity="warning"
        ))

# Create and test model
complex_model = SEQIAHRWithValidation()
print(f"Created {complex_model.model_type} model")
print(f"Parameters: {list(complex_model.parameter_specs.keys())}")
print(f"Constraints: {len(complex_model.model_constraints)}")

Created SEQIAHR model
Parameters: ['beta', 'p', 'chi', 'gamma', 'mu']
Constraints: 2

[14]:

# Test complex model validation
params_valid = {
    'beta': 0.3,
    'p': 0.4,
    'chi': 0.7,
    'gamma': 0.1,
    'mu': 0.01
}

print("Validating complex model with valid parameters:")
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    complex_model.validate_parameters(params_valid)
    print(f"  ✓ Validation passed with {len(w)} warnings")

Validating complex model with valid parameters:
  ✓ Validation passed with 0 warnings

[15]:

# Test with probability out of bounds
params_invalid = {
    'beta': 0.3,
    'p': 1.5,  # Invalid: > 1
    'chi': 0.7,
    'gamma': 0.1,
    'mu': 0.01
}

print("Testing with probability out of bounds (p=1.5):")
try:
    complex_model.validate_parameters(params_invalid)
    print("  ✗ Should have raised error")
except ValidationError as e:
    print(f"  ✓ Caught error: {str(e)[:80]}...")

Testing with probability out of bounds (p=1.5):
  ✓ Caught error: Parameter 'p' value 1.5 exceeds maximum bound 1
Parameter 'p' value 1.5 violates...

7. Backward Compatibility

Existing models without rich specs still work with simple validation.

[16]:

# Legacy-style model (backward compatible)
class LegacyModel(BaseModel):
    def __init__(self):
        super().__init__()
        self.model_type = "Legacy"
        # Old-style simple dicts
        self.parameters = {"alpha": r"$\alpha$", "delta": r"$\delta$"}
        self.state_variables = {"X": "Variable X", "Y": "Variable Y"}

legacy = LegacyModel()
print(f"Created legacy model with {len(legacy.parameters)} parameters")

# Simple validation still works
legacy.validate_parameters({'alpha': 0.5, 'delta': 0.2})
print("✓ Simple validation works")

try:
    legacy.validate_parameters({'alpha': 0.5})
    print("✗ Should have caught missing parameter")
except ValidationError:
    print("✓ Simple validation catches missing parameters")

try:
    legacy.validate_parameters({'alpha': -0.5, 'delta': 0.2})
    print("✗ Should have caught negative parameter")
except ValidationError:
    print("✓ Simple validation catches negative parameters")

Created legacy model with 2 parameters
✓ Simple validation works
✓ Simple validation catches missing parameters
✓ Simple validation catches negative parameters

8. Generating Documentation

[17]:

def generate_parameter_docs(model):
    """Generate markdown documentation from parameter specs."""
    if not model.parameter_specs:
        return "No parameter specifications defined."

    doc = f"# {model.model_type} Model Parameters\n\n"

    doc += "## Parameters\n\n"
    for name, spec in model.parameter_specs.items():
        doc += f"### {spec.symbol} ({name})\n\n"
        doc += f"{spec.description}\n\n"
        if spec.bounds:
            min_val, max_val = spec.bounds
            doc += f"- **Bounds**: {min_val or '−∞'} to {max_val or '∞'}\n"
        if spec.units:
            doc += f"- **Units**: {spec.units}\n"
        if spec.typical_range:
            doc += f"- **Typical range**: {spec.typical_range[0]} to {spec.typical_range[1]}\n"
        if spec.constraints:
            doc += f"- **Constraints**: {', '.join(spec.constraints)}\n"
        doc += "\n"

    if model.model_constraints:
        doc += "## Model Constraints\n\n"
        for constraint in model.model_constraints:
            severity_str = f" ({constraint.severity})" if constraint.severity != "error" else ""
            doc += f"- {constraint.description}{severity_str}\n"
            doc += f"  - Expression: `{constraint.expression}`\n\n"

    return doc

# Generate documentation for SIR model
print(generate_parameter_docs(model))

# SIR Model Parameters

## Parameters

### $\beta$ (beta)

Transmission rate

- **Bounds**: −∞ to ∞
- **Units**: 1/day
- **Typical range**: 0.1 to 1.0
- **Constraints**: value > 0

### $\gamma$ (gamma)

Recovery rate (1 / average infectious period)

- **Bounds**: −∞ to ∞
- **Units**: 1/day
- **Typical range**: 0.05 to 0.5
- **Constraints**: value > 0

## Model Constraints

- R0 > 1 required for epidemic spread (warning)
  - Expression: `beta / gamma > 1`

Summary

This notebook demonstrated:

  1. ParameterSpec: Define parameters with bounds, constraints, units, and documentation

  2. VariableSpec: Define state variables with validation rules

  3. ModelConstraint: Cross-parameter constraints with severity levels

  4. Constraint Language: Flexible expression evaluation

  5. Validation Errors: Clear error messages for constraint violations

  6. Warning Constraints: Soft constraints that warn but don’t fail

  7. Backward Compatibility: Existing models work unchanged

  8. Documentation Generation: Automatic documentation from specs