6. Automated reliability testing

Error-handling code is still code, and code that isn't exercised regularly breaks quietly. The whole point of the retry, categorize, and log pipeline is that users never see it firing, which means you'll never notice if something about it stops working. Tests fix that. You simulate every failure the network can throw at you from inside a deterministic, millisecond-scale test run, and you find out the instant a regression sneaks in instead of waiting for a support ticket.

Why automated testing matters

Manual testing is slow, inconsistent, and incomplete. You can't reliably simulate network failures or reproduce race conditions by hand. How do you test that a three-second timeout triggers the right retry logic? Or that a thousand simultaneous failures don't create a thundering herd? Automated tests run in seconds, catch edge cases you'd never think to reproduce manually, and prevent regressions when you refactor. They're the only way you find out the error-handling pipeline still works before a user does.

What to test

Your error handling system has four testable components:

Categorization logic: Does categorize_error() map exceptions to the correct categories?
Message composition: Does each category generate proper three-part messages with required fields?
Retry behavior: Does retry logic respect max attempts, use exponential backoff, and skip non-transient errors?
Integration: Does the complete flow work end-to-end with mocked API failures?

Testing each component independently makes tests fast, focused, and easy to debug. Integration tests verify they work together correctly.

Setting up pytest with mocks

pytest provides everything needed for testing error handling. Install it and create a test file:

Terminal

# Install pytest into your project's virtual environment
pip install pytest pytest-mock

# Create the test file
touch test_weather_dashboard.py

# Run the tests
pytest test_weather_dashboard.py -v

The pytest-mock plugin lets you simulate network failures, timeouts, and API responses without making real requests. This makes tests fast (no network wait), reliable (no dependence on external services), and repeatable (same behavior every run).

Complete test suite

Here's a comprehensive test suite that covers all error handling components. Create this as test_weather_dashboard.py:

test_weather_dashboard.py

import pytest
from unittest.mock import Mock, patch
from requests.exceptions import Timeout, ConnectionError, HTTPError
from weather_dashboard import (
    categorize_error,
    compose_error_message,
    validate_city_name,
    retry_with_backoff,
    find_location,
    get_weather
)

# ==================== CATEGORIZATION TESTS ====================

def test_categorize_empty_user_input():
    """Empty city name should categorize as user_input."""
    exception = ValueError("City name cannot be empty")
    category, error_type, context = categorize_error(exception)
    assert category == "user_input"
    assert error_type == "empty"

def test_categorize_whitespace_input():
    """Whitespace-only input should categorize as user_input."""
    exception = ValueError("City name cannot be empty")
    category, error_type, context = categorize_error(exception)
    assert category == "user_input"

def test_categorize_too_long_input():
    """City name over 100 chars should categorize as user_input."""
    exception = ValueError("City name too long (maximum 100 characters)")
    category, error_type, context = categorize_error(exception)
    assert category == "user_input"
    assert error_type == "too_long"

def test_categorize_invalid_chars():
    """Invalid characters should categorize as user_input."""
    exception = ValueError("City name contains invalid characters")
    category, error_type, context = categorize_error(exception)
    assert category == "user_input"
    assert error_type == "invalid_chars"

def test_categorize_timeout():
    """Network timeout should categorize as transient."""
    exception = Timeout("Request timed out")
    category, error_type, context = categorize_error(exception)
    assert category == "transient"
    assert error_type == "timeout"

def test_categorize_connection_error():
    """Connection errors should categorize as transient."""
    exception = ConnectionError("Failed to establish connection")
    category, error_type, context = categorize_error(exception)
    assert category == "transient"
    assert error_type == "connection"

def test_categorize_rate_limit():
    """429 rate limit should categorize as transient with retry context."""
    response = type('Response', (), {
        'status_code': 429,
        'headers': {'Retry-After': '60'}
    })()
    exception = HTTPError(response=response)
    
    category, error_type, context = categorize_error(exception, response)
    
    assert category == "transient"
    assert error_type == "rate_limit"
    assert context["retry_seconds"] == "60"

def test_categorize_server_error_500():
    """500 server error should categorize as transient."""
    response = type('Response', (), {'status_code': 500})()
    exception = HTTPError(response=response)
    category, error_type, context = categorize_error(exception, response)
    assert category == "transient"
    assert error_type == "server_error"

def test_categorize_server_error_503():
    """503 service unavailable should categorize as transient."""
    response = type('Response', (), {'status_code': 503})()
    exception = HTTPError(response=response)
    category, error_type, context = categorize_error(exception, response)
    assert category == "transient"

def test_categorize_key_error():
    """KeyError for 'results' should categorize as not_found."""
    exception = KeyError('results')
    category, error_type, context = categorize_error(exception)
    assert category == "not_found"
    assert error_type == "city"

def test_categorize_404():
    """404 response should categorize as not_found."""
    response = type('Response', (), {'status_code': 404})()
    exception = HTTPError(response=response)
    category, error_type, context = categorize_error(exception, response)
    assert category == "not_found"

def test_categorize_unknown_error():
    """Unexpected exceptions should categorize as unknown."""
    exception = RuntimeError("Something weird happened")
    category, error_type, context = categorize_error(exception)
    assert category == "unknown"
    assert error_type == "general"

# ==================== MESSAGE COMPOSITION TESTS ====================

def test_message_has_three_parts_empty_input():
    """Empty input message should have three parts."""
    message = compose_error_message("user_input", "empty")
    lines = message.split('\n')
    assert len(lines) == 3

def test_message_has_three_parts_not_found():
    """Not found message should have three parts."""
    message = compose_error_message("not_found", "city",
                                   city_name="Lndon",
                                   suggestions="London, Dublin")
    lines = message.split('\n')
    assert len(lines) == 3
    assert "Lndon" in lines[0]

def test_message_quotes_user_input():
    """Messages should quote back user input."""
    message = compose_error_message("not_found", "city",
                                   city_name="Lndon",
                                   suggestions="London, Dublin")
    assert "Lndon" in message

def test_message_includes_retry_time():
    """Rate limit messages should include retry time."""
    message = compose_error_message("transient", "rate_limit",
                                   retry_seconds="60")
    assert "60" in message
    assert "seconds" in message.lower()

def test_message_transient_no_technical_jargon():
    """Transient error messages should avoid technical terms."""
    message = compose_error_message("transient", "timeout")
    
    # Should NOT contain technical terms
    assert "KeyError" not in message
    assert "Traceback" not in message
    assert "Exception" not in message
    
    # SHOULD contain friendly language
    assert any(word in message.lower() for word in 
               ["trouble", "try", "moment", "temporary"])

def test_message_unknown_gives_actionable_guidance():
    """Unknown error messages should still be actionable."""
    message = compose_error_message("unknown", "general")
    assert "try again" in message.lower()
    assert len(message.split('\n')) == 3

# ==================== RETRY LOGIC TESTS ====================

@patch('weather_dashboard.requests.get')
def test_retry_succeeds_on_second_attempt(mock_get, mocker):
    """Should retry transient failures and succeed eventually."""
    # Mock first call fails, second succeeds
    mock_get.side_effect = [
        Timeout("First attempt fails"),
        type('Response', (), {
            'status_code': 200,
            'raise_for_status': lambda self: None,
            'json': lambda self: {"results": [{"latitude": 51.5, "longitude": -0.1}]}
        })()
    ]
    
    mocker.patch('time.sleep')
    
    def test_func():
        response = mock_get()
        response.raise_for_status()
        return response.json()
    
    result, error = retry_with_backoff(test_func, max_attempts=3)
    
    assert error is None
    assert result is not None
    assert mock_get.call_count == 2

@patch('weather_dashboard.requests.get')
def test_retry_respects_max_attempts(mock_get, mocker):
    """Should stop after max_attempts failures."""
    mock_get.side_effect = Timeout("Always fails")
    mocker.patch('time.sleep')
    
    def test_func():
        response = mock_get()
        return response.json()
    
    result, error = retry_with_backoff(test_func, max_attempts=3)
    
    assert result is None
    assert error is not None
    assert mock_get.call_count == 3

@patch('weather_dashboard.requests.get')
def test_no_retry_for_user_input_error(mock_get, mocker):
    """Should not retry user_input category errors."""
    mocker.patch('time.sleep')
    
    def test_func():
        raise ValueError("City name cannot be empty")
    
    result, error = retry_with_backoff(test_func, max_attempts=3)
    
    assert result is None
    assert "city name" in error.lower()

@patch('weather_dashboard.requests.get')
def test_retry_uses_exponential_backoff(mock_get, mocker):
    """Should wait with exponential backoff between retries."""
    mock_get.side_effect = [
        Timeout("Fail"),
        Timeout("Fail again"),
        type('Response', (), {'json': lambda self: {"success": True}})()
    ]
    
    sleep_spy = mocker.patch('time.sleep')
    
    def test_func():
        response = mock_get()
        return response.json()
    
    result, error = retry_with_backoff(test_func, max_attempts=3, base_delay=1.0)
    
    sleep_calls = [call[0][0] for call in sleep_spy.call_args_list]
    assert len(sleep_calls) == 2
    assert sleep_calls[0] < 2.0
    assert sleep_calls[1] >= 2.0

# ==================== INTEGRATION TESTS ====================

@patch('weather_dashboard.requests.get')
def test_end_to_end_city_not_found(mock_get):
    """Complete flow for city not found."""
    mock_get.return_value.json.return_value = {
        "results": []
    }
    mock_get.return_value.status_code = 200
    mock_get.return_value.raise_for_status = Mock()
    
    result, error = find_location("Lndon")
    
    assert result is None
    assert "Lndon" in error
    assert "couldn't find" in error.lower()

def test_categorization_drives_message_selection():
    """Verify categorization determines which message template is used."""
    test_cases = [
        (ValueError("City name cannot be empty"), "user_input", "empty", {}),
        (Timeout("timeout"), "transient", "timeout", {}),
        (KeyError("results"), "not_found", "city", {
            "city_name": "Lndon",
            "suggestions": "London, Dublin"
        }),
        (RuntimeError("unknown"), "unknown", "general", {})
    ]
    
    for exception, expected_category, expected_type, message_context in test_cases:
        category, error_type, context = categorize_error(exception)
        assert category == expected_category
        assert error_type == expected_type
        
        message_context.update(context)
        message = compose_error_message(category, error_type, **message_context)
        assert len(message.split('\n')) == 3

This comprehensive test suite covers:

Categorization (12 tests): Verifies all exception types map to correct categories
Message composition (6 tests): Ensures messages follow three-part pattern and avoid technical jargon
Retry logic (4 tests): Tests exponential backoff, max attempts, and selective retry
Integration (2 tests): End-to-end flows with mocked API responses

Running the test suite

Run your complete test suite with verbose output:

Terminal

pytest test_weather_dashboard.py -v

# Expected output:
test_categorize_empty_user_input PASSED                       [  4%]
test_categorize_whitespace_input PASSED                       [  8%]
test_categorize_too_long_input PASSED                         [ 12%]
test_categorize_invalid_chars PASSED                          [ 16%]
test_categorize_timeout PASSED                                [ 20%]
test_categorize_connection_error PASSED                       [ 25%]
test_categorize_rate_limit PASSED                             [ 29%]
test_categorize_server_error_500 PASSED                       [ 33%]
test_categorize_server_error_503 PASSED                       [ 37%]
test_categorize_key_error PASSED                              [ 41%]
test_categorize_404 PASSED                                    [ 45%]
test_categorize_unknown_error PASSED                          [ 50%]
test_message_has_three_parts_empty_input PASSED               [ 54%]
test_message_has_three_parts_not_found PASSED                 [ 58%]
test_message_quotes_user_input PASSED                         [ 62%]
test_message_includes_retry_time PASSED                       [ 66%]
test_message_transient_no_technical_jargon PASSED             [ 70%]
test_message_unknown_gives_actionable_guidance PASSED         [ 75%]
test_retry_succeeds_on_second_attempt PASSED                  [ 79%]
test_retry_respects_max_attempts PASSED                       [ 83%]
test_no_retry_for_user_input_error PASSED                     [ 87%]
test_retry_uses_exponential_backoff PASSED                    [ 91%]
test_end_to_end_city_not_found PASSED                         [ 95%]
test_categorization_drives_message_selection PASSED           [100%]

==================== 24 passed in 0.23s ====================

Your complete test suite runs in under a second and verifies every component of your error handling system works correctly. Change code freely - tests catch regressions before deployment.