Recursive Schemas with Definitions

In pydantic-core, defining recursive data structures—such as trees, linked lists, or self-referencing models—requires a specific strategy to avoid infinite recursion during schema construction. Because Python evaluates objects eagerly, nesting a schema directly within itself would lead to a RecursionError before the schema is even built.

To solve this, the codebase implements a "definitions and references" pattern using DefinitionsSchema and DefinitionReferenceSchema. This approach separates the definition of a schema from its usage, allowing schemas to refer to themselves or each other by a symbolic string identifier.

The Definitions Registry

The core of this system is the DefinitionsSchema, defined in pydantic-core/python/pydantic_core/core_schema.py. It acts as a container that holds:

1. schema: The primary entry point for validation.
2. definitions: A list of CoreSchema objects that can be referenced by name.

Each schema within the definitions list must have a ref attribute. This ref is a unique string identifier that other parts of the schema use to point back to it.

Basic Shared Reference

Beyond recursion, definitions are useful for reusability. If multiple fields in a schema use the same complex validation logic, you can define it once in the definitions list and reference it multiple times, reducing the overall schema size.

from pydantic_core import SchemaValidator, core_schema

# Define a shared integer schema with the ref 'foobar'
schema = core_schema.definitions_schema(
    # The main schema: a list of references to 'foobar'
    core_schema.list_schema(core_schema.definition_reference_schema('foobar')),
    [
        # The registry: contains the actual integer schema
        core_schema.int_schema(ref='foobar')
    ],
)

v = SchemaValidator(schema)
assert v.validate_python([1, 2, '3']) == [1, 2, 3]

Implementing Recursive Structures

To create a recursive structure, a schema in the definitions list includes a DefinitionReferenceSchema that points to its own ref.

Recursive Trees

A common use case is a "Branch" or "Node" structure where each item can contain another item of the same type. The following example from pydantic-core/tests/validators/test_definitions_recursive.py demonstrates a recursive typed-dict:

from pydantic_core import SchemaValidator, core_schema

v = SchemaValidator(
    core_schema.definitions_schema(
        # Entry point: start by looking up the 'Branch' definition
        core_schema.definition_reference_schema('Branch'),
        [
            {
                'type': 'typed-dict',
                'ref': 'Branch',  # This ID is used for recursion
                'fields': {
                    'name': {'type': 'typed-dict-field', 'schema': {'type': 'str'}},
                    'sub_branch': {
                        'type': 'typed-dict-field',
                        'schema': {
                            'type': 'default',
                            'schema': {
                                'type': 'nullable',
                                # Recursive reference back to 'Branch'
                                'schema': core_schema.definition_reference_schema('Branch'),
                            },
                            'default': None,
                        },
                    },
                },
            }
        ],
    )
)

# Validation handles the nested structure
assert v.validate_python({'name': 'root', 'sub_branch': {'name': 'b1'}}) == (
    {'name': 'root', 'sub_branch': {'name': 'b1', 'sub_branch': None}}
)

Recursive Lists

You can also define recursive sequences. In this example from the definition_reference_schema docstring, a list schema is defined such that its items are also instances of that same list schema:

from pydantic_core import SchemaValidator, core_schema

schema_definition = core_schema.definition_reference_schema('list-schema')
schema = core_schema.definitions_schema(
    schema=schema_definition,
    definitions=[
        core_schema.list_schema(items_schema=schema_definition, ref='list-schema'),
    ],
)

v = SchemaValidator(schema)
# Validates a list containing an empty list (which matches the recursive definition)
assert v.validate_python([()]) == [[]]

Runtime Behavior and Constraints

While DefinitionsSchema allows the schema to be recursive, pydantic-core still protects against infinite loops in the input data during validation.

Circular Data Detection

If you provide data that contains a circular reference (e.g., an object that points to itself), the validator will detect the cycle and raise a ValidationError with a recursion_loop error type. This prevents the validation process from hanging or crashing with a stack overflow.

Schema Integrity

The schema_ref in a DefinitionReferenceSchema must correspond to a ref defined within the definitions list of the enclosing DefinitionsSchema. If a reference is made to a non-existent ref, pydantic-core will raise a SchemaError during the initialization of the SchemaValidator.

Internal Usage

In the broader Pydantic ecosystem, the pydantic._internal._generate_schema module automatically utilizes definitions_schema when it encounters recursive Python types (like class Node: child: Optional['Node']). It collects all discovered references and bundles them into a single top-level DefinitionsSchema block before passing it to pydantic-core.