Part 2: Action Operator Formalization — Definitions, Families, and Implementation¶
Series: Action Operators in Generalized Reinforcement Learning
Part: 2 of 4
Audience: Self-reference and interview prep
Prerequisites: Part 1 (The GRL0 Gap)
1. The Core Shift: From Selection to Construction¶
Part 1 identified the gap: GRL0 had a rich value landscape over parametric actions but no formal mechanism for translating parameters into state transformations. The solution is a conceptual shift in what "taking an action" means:
- Classical RL: The agent selects from a predefined menu. The environment interprets the selection and produces the next state.
- GRL (operator framework): The agent constructs a function that transforms the state. The environment's role is reduced to adding noise.
This shift elevates the agent from a decision-maker to a dynamics constructor. The policy doesn't output "which action" but "how to transform the current state."
2. The Three Definitions¶
The operator framework rests on three interlocking definitions that together close the gap identified in Part 1.
Definition 1: Action Operator¶
An action operator is a measurable mapping \(\hat{O}: \mathcal{S} \to \mathcal{S}\) that transforms states directly.
In practice, we restrict to smooth or Lipschitz operators for well-behaved dynamics. Each operator carries an energy functional \(E(\hat{O}) \geq 0\) that measures transformation magnitude, enabling least-action regularization.
What this means: An action is no longer a symbol or a parameter vector. It is a function — a complete specification of how every state in \(\mathcal{S}\) would be transformed. The agent doesn't say "apply force 2.5 to joint 3"; it says "here is a function that maps every possible state to its successor."
The transition rule becomes:
The operator directly produces the next state. The environment contributes only stochastic perturbation \(\xi\) representing unmodeled dynamics or noise. In the deterministic limit (\(\sigma \to 0\)), the agent has full control over the transition.
Definition 2: Operator Generator¶
An operator generator is a mapping:
where \(\Theta \subseteq \mathbb{R}^d\) is the parameter space. For fixed \(\theta \in \Theta\):
The set \(\{\hat{O}_\theta : \theta \in \Theta\}\) defines a parametric operator manifold — a smooth surface embedded in the (generally infinite-dimensional) operator space \(\mathcal{O}\).
What this means: This is the missing bridge from Part 1. GRL0 had continuous parameters \(\theta\) but no formal way to convert them into state transformations. The generator \(\Phi\) does exactly this: given any \(\theta\), it produces a concrete operator that can be applied to any state.
Different choices of \(\Phi\) yield different operator families (see Section 3 below). The generator is the design choice that determines what class of transformations the agent can express.
In neural implementations, \(\Phi\) is typically a neural network that takes \((\theta, s)\) as input and outputs \(s'\). The parameters \(\theta\) may themselves be generated by a policy network.
Definition 3: Operator Policy¶
An operator policy is a mapping:
that assigns to each state a distribution over operator parameters. For deterministic policies: \(\pi: \mathcal{S} \to \Theta\).
The full chain:
What this means: The policy is a meta-function — it outputs not a primitive value but a specification for a state transformation. The architecture has two components:
- Policy network \(\pi_\psi\): maps states to operator parameters
- Operator generator \(\Phi\): maps parameters and states to transformed states
This separation is powerful. The policy network can be a standard MLP or transformer. The operator generator encodes the structure of allowable transformations. Different tasks might share the same generator family (e.g., field operators) with different learned policies.
How the Three Definitions Interlock¶
flowchart TB
S(["<b>State</b><br/>s ∈ 𝒮"])
D3["<b>Definition 3 · Operator Policy</b><br/>π<sub>ψ</sub> : 𝒮 → Δ(Θ)<br/><i>how a state chooses parameters</i>"]
D2["<b>Definition 2 · Operator Generator</b><br/>Φ : Θ × 𝒮 → 𝒮<br/><i>how parameters become an operator</i>"]
D1["<b>Definition 1 · Action Operator</b><br/>Ô<sub>θ</sub> : 𝒮 → 𝒮<br/><i>what an action IS</i>"]
SP(["<b>Next State</b><br/>s′ = Ô<sub>θ</sub>(s) + ξ"])
S --> D3
D3 -- "θ ∈ Θ" --> D2
D2 -- "Ôθ = Φ(θ, ·)" --> D1
D1 -- "apply to s, + noise ξ" --> SP
style S fill:#e3f2fd,stroke:#1976d2,stroke-width:3px,color:#000
style D3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px,color:#000
style D2 fill:#fff59d,stroke:#fbc02d,stroke-width:3px,color:#000
style D1 fill:#ffcc80,stroke:#f57c00,stroke-width:3px,color:#000
style SP fill:#c8e6c9,stroke:#388e3c,stroke-width:3px,color:#000Definition 1 says what an action is. Definition 2 says how to construct one from parameters. Definition 3 says how to choose parameters given the current state. Together, they provide a complete, differentiable pipeline from state to next state — no discretization required.
3. Operator Families: Concrete Realizations¶
Each operator family is defined by a specific form of the generator \(\Phi(\theta, s)\). The choice of family encodes structural assumptions about what kinds of transformations are relevant for the task.
3.1 Affine Operators¶
Parameters: \(\theta = (\text{vec}(A), b) \in \mathbb{R}^{n^2 + n}\)
Generator: \(\Phi(\theta, s) = A_\theta s + b_\theta\)
Energy: \(E(\hat{O}) = \|A - I\|_F^2 + \|b\|_2^2\)
Why it matters: This is the bridge to classical RL. When \(A = I\), the operator reduces to \(\hat{O}(s) = s + b\), which is exactly the displacement model in standard continuous RL — the "action" is a vector added to the state. Affine operators generalize this by allowing the transformation to depend on the current state through the matrix \(A\).
When to use: Linear control problems, systems where the dynamics are approximately linear near the operating point, or as a baseline to validate the framework.
Recovery of classical RL: Setting \(\mathcal{O} = \{s \mapsto s + b_1, \ldots, s \mapsto s + b_k\}\) for a finite set of displacements recovers a discrete-action MDP. This is the formal proof that standard RL is a special case of GRL.
3.2 Field Operators¶
Parameters: \(\theta\) = weights of a neural network \(F_\theta: \mathcal{S} \to \mathcal{S}\)
Generator: \(\Phi(\theta, s) = s + F_\theta(s)\)
Energy: \(E(\hat{O}) = \mathbb{E}_s[\|F_\theta(s)\|^2]\)
Why it matters: This is the operator family most directly connected to GRL0's reinforcement field concept. In GRL0, the reinforcement field \(\nabla E\) was the value gradient — it pointed toward better configurations. In the operator framework, the field is the action: the agent generates a vector field \(F_\theta\) that specifies the displacement at every state.
The key difference: in GRL0, the field emerged from the value function and was used to select action parameters. Here, the field is the operator itself — it directly defines the state transition.
Connection to neural ODEs: The field operator \(\hat{O}(s) = s + \Delta t \cdot F_\theta(s)\) is an Euler step of the ODE \(\dot{s} = F_\theta(s)\). For small \(\Delta t\), this approximates the flow of a dynamical system. The agent is learning an infinitesimal generator of dynamics.
When to use: Navigation, flow control, systems where smooth continuous dynamics are natural.
3.3 Potential Field Operators¶
Parameters: \(\theta\) = weights of a scalar potential network \(\phi_\theta: \mathcal{S} \to \mathbb{R}\)
Generator: \(\Phi(\theta, s) = s - \eta \nabla_s \phi_\theta(s)\)
Energy: \(E(\hat{O}) = \mathbb{E}_s[\|\nabla \phi(s)\|^2]\)
Why it matters: This is a conservative field operator — the displacement is always the gradient of a scalar potential, so the induced dynamics are curl-free. The agent learns a potential landscape, and the action is gradient descent on that landscape.
Connection to physics: This is exactly how conservative forces work in classical mechanics. The potential \(\phi\) plays the role of a learned energy landscape, and the operator moves the state downhill on that landscape.
Implementation note: The gradient \(\nabla_s \phi\) is computed via torch.autograd.grad, making this fully differentiable. The potential network outputs a scalar, and the gradient gives the displacement vector — the operator never explicitly constructs the vector field.
When to use: Systems with conservative dynamics (gravity, electrostatics), or when you want the operator to have provably convergent dynamics (gradient flows always decrease the potential).
3.4 Kernel/RBF Operators¶
Parameters: \(\theta = (c_1, \ldots, c_M, d_1, \ldots, d_M, w_1, \ldots, w_M)\)
Generator: Localized displacement effects centered at learned locations \(c_i\) with directions \(d_i\) and weights \(w_i\).
Energy: \(E(\hat{O}) = \sum_i \|d_i\|^2\) (magnitude of displacement directions)
Why it matters: This creates localized operator effects — attractors and repellers at specific locations in state space. It directly bridges to GRL0's particle concept: in GRL0, particles defined the value function; here, particles define the operator.
Connection to GRL0: In GRL0, particles \((x_i, w_i)\) stored experience and shaped the energy landscape \(Q^+ = \sum w_i k(x, x_i)\). In the RBF operator, learned centers \((c_i, d_i)\) shape the action field. The mathematical form is identical — the interpretation shifts from "evidence about value" to "specification of transformation."
When to use: Navigation with obstacles (centers as repellers), multi-modal control (different centers for different behavioral modes), spatially varying dynamics.
3.5 Attention-Based Kernel Operators¶
where \(\alpha_i(s) = \text{softmax}(f_\text{attn}(s))_i\) and \(T_i\) are basis transformations.
Parameters: \(\theta\) = attention network weights + basis transform parameters
Generator: State-dependent weighted mixture of basis operators.
Energy: Entropy of attention weights (encourages committing to specific transformations).
Why it matters: This is the most expressive implemented family. The operator adapts its behavior based on the current state — in one region of state space it might apply transformation \(T_1\), in another it might blend \(T_2\) and \(T_3\). This is a kernel operator in the generalized sense: the "kernel" is the attention function that determines which basis transformations are active.
When to use: Complex tasks with multiple behavioral modes, tool-use scenarios, tasks requiring different strategies in different regions of state space.
3.6 Hamiltonian Operators (Planned)¶
where \(J\) is the symplectic matrix and \(H\) is the Hamiltonian.
Why it matters: Always invertible, energy-conserving, preserves symplectic structure. Natural for physical systems where energy conservation is a hard constraint. The symplectic group \(Sp(2n)\) provides a Lie group structure for the operator space.
4. Algebraic Structure: Why It Matters¶
Operators aren't just individual objects — they compose, and the algebraic structure of composition has deep implications for how GRL works.
4.1 Composition and Monoid Structure¶
Operators compose naturally: \((\hat{O}_2 \circ \hat{O}_1)(s) = \hat{O}_2(\hat{O}_1(s))\). With the identity operator \(I(s) = s\), the triple \((\mathcal{O}, \circ, I)\) forms a monoid — closure, associativity, identity.
Why it matters for RL: Sequential application of operators is just composition. A trajectory of \(K\) timesteps is the composition \(\hat{O}_K \circ \cdots \circ \hat{O}_1\). This means a skill is a fixed composition of operators, and hierarchical RL is meta-policy over operator compositions.
4.2 Energy of Composed Operators¶
Composition energy is subadditive:
This bounds the complexity of composed operators and connects to the least-action principle: the cheapest way to achieve a goal is a sequence of individually cheap operators.
4.3 Invertibility and Group Structure¶
For invertible operator families (affine with \(\det(A) \neq 0\), Hamiltonian always), we get a group structure — operators can be undone. This enables:
- Reversible dynamics (undo the last action)
- Time-reversal symmetry
- Cycle consistency as a regularizer
4.4 Lie Group Structure¶
When the operator group is also a smooth manifold, we get a Lie group. The Lie algebra \(\mathfrak{g} = T_I \mathcal{G}\) (tangent space at identity) provides a linear parameterization of a nonlinear operator space:
Architectural implication: The policy network can output a Lie algebra element \(X\) (which lives in a vector space — easy to parameterize), and exponentiation produces the operator (which lives in the nonlinear group). This is exactly how rotations work: output an angle (Lie algebra \(\mathfrak{so}(2) = \mathbb{R}\)), exponentiate to get a rotation matrix (Lie group \(SO(2)\)).
5. The Generalized Bellman Equation¶
With operators formalized, the Bellman equation generalizes naturally. The action-value function is now defined over operator parameters:
where: - \(R(s, s')\) is the reward for transitioning from \(s\) to \(s'\) - \(E(\hat{O}_\theta)\) is the operator energy (least-action penalty) - \(\gamma\) is the discount factor - \(V(s') = \max_\theta Q(s', \theta)\) is the state value
Recovery of classical Bellman: When \(\mathcal{O}\) is restricted to a finite set \(\{\hat{O}_{a_1}, \ldots, \hat{O}_{a_k}\}\) and \(\lambda = 0\) (no energy regularization), this reduces to:
which is the standard Bellman equation. GRL strictly generalizes classical RL.
The energy term is what makes GRL distinct from just "continuous-action RL." The penalty \(-\lambda E(\hat{O}_\theta)\) in the effective reward creates a preference for simple operators. Among operators that achieve the same reward, the agent prefers the one with minimal energy — the one that transforms the state as little as necessary. This is the RL analogue of the principle of least action in physics.
6. The Operator Generator Network: Implementation¶
In practice, the policy doesn't directly output an operator (which is a function — infinite-dimensional). Instead, it outputs parameters \(\theta\) that are fed to a generator. Two architectures have been implemented:
6.1 Parameter Prediction¶
The OperatorGenerator class uses a standard architecture:
- Encoder: MLP maps state \(s\) to hidden representation \(h\)
- Parameter head: Linear layer maps \(h\) to flat parameter vector \(\theta\)
- Operator instantiation: \(\theta\) is reshaped and loaded into the target operator class
Initialization: The parameter head is initialized to zeros, so the initial operator is near-identity. This ensures stable early training and aligns with least-action regularization.
6.2 Hypernetwork Generation¶
The HypernetworkGenerator is more expressive. Instead of predicting parameters for a fixed operator class, a hypernetwork generates the weights of a target operator network directly:
- Hypernetwork: Maps state \(s\) to a flat vector of target network weights
- Target network: Uses generated weights to compute \(F(s)\) (the displacement field)
- Residual: \(s' = s + F(s)\)
Per-sample operators: The hypernetwork naturally produces per-sample operators via batched matrix multiplication. Each state in a batch gets its own operator network — different states generate different transformations. This is a key advantage over the parameter prediction approach, which averages over the batch.
7. Connecting Back to GRL0¶
The operator framework doesn't invalidate GRL0 — it completes it. Here's how the original concepts map to the new formalism:
| GRL0 Concept | Operator Framework | Relationship |
|---|---|---|
| Parametric action \(\theta\) | Operator parameter \(\theta\) | Same object, now with formal semantics |
| Fitness/energy \(E(s, \theta)\) | Operator energy \(E(\hat{O}_\theta)\) | Value-side → structure-side |
| Reinforcement field \(\nabla E\) | Operator gradient flow | Field guides optimization over \(\theta\) |
| GP value function | Neural operator value | Same role, different function class |
| Augmented space \((s, \theta)\) | Operator manifold \(\{\hat{O}_\theta\}\) | Geometric interpretation preserved |
| Experience particles | Operator evaluations | Same sampling philosophy |
| Discretized \(\Theta\) | Continuous \(\Theta\) via \(\Phi\) | Gap closed |
The key difference: in GRL0, \(\theta\) parameterized the value landscape but not the dynamics. In the operator framework, \(\theta\) parameterizes the transformation itself via the generator \(\Phi\). The reinforcement field can still guide the search over \(\theta\), but now executing \(\theta\) is well-defined at every point — not just at discretized bins.
8. Summary¶
The action operator formalization provides three things that were missing from GRL0:
-
A formal answer to "what is an action?" — It's a measurable function \(\hat{O}: \mathcal{S} \to \mathcal{S}\), not a symbol or parameter.
-
A formal bridge from parameters to transformations — The operator generator \(\Phi(\theta, s)\) converts continuous parameters into executable state transformations, eliminating the need for discretization.
-
Rich algebraic structure — Composition, invertibility, Lie group structure, and energy functionals provide a mathematical framework for reasoning about action sequences, hierarchical control, and regularization.
Together, these enable a fully continuous, differentiable pipeline from state observation to state transformation — the original promise of GRL0, now realized.
Next: Part 3: From Fixed Kernels to Learned Kernels — extending the RKHS machinery via Mercer's theorem, the metric learning interpretation, and learned feature maps.
Related documents:
- Operator base class: src/grl/operators/base.py
- Operator families: src/grl/operators/{affine,field,kernel}.py
- Operator generator: src/grl/policies/generator.py