Physical Firmware

0. The Problem Nobody Is Asking Loudly Enough

A toddler who has never thrown a baseball will, after a few minutes of play, develop a reasonable throw. A robot trained with state-of-the-art methods needs thousands of demonstrations to learn a comparable skill — and still fails when the ball changes size.

The toddler isn't smarter. The toddler has something the robot doesn't: priors. Not physics-class priors. The toddler has never heard of Newton's laws. But they know, in a deep embodied way, that things fall when released, objects don't pass through each other, heavy things are harder to lift, spinning things behave differently. This is not knowing $F = ma$. It is something more primitive and more powerful — because it generalizes instantly to any new situation without derivation.

The answer I'm exploring: every robot should be initialized with what I call Physical Firmware — a pretrained, structured world model that encodes fundamental physical understanding before any task-specific training begins. Not behavioral pretraining. Not a hard-coded simulator. A learned model with physical structure built into its architecture, that gives any robot a grounded understanding of how the world works.

When a toddler learns to throw a ball, they are not learning physics. They are fitting a new skill onto a physical foundation that already exists. That is the goal.

1. System Architecture

Physical Firmware is organized as a four-layer stack. The bottom two layers constitute the firmware itself and are trained once. The upper two are adapted per-robot and per-task.

Layers 0 and 1 are the core contribution. They are trained once on diverse physical interaction data and transfer across robots, environments, and tasks without modification. Layers 2 and 3 are the only things a new customer needs to train.

2. The Physics Core: Hamiltonian Mechanics

Most people learn physics through Newton's second law: $\mathbf{F} = m\mathbf{a}$. Identify all forces, write an equation for each, solve the system. This works but scales poorly — for a robot arm with seven joints you need to track forces and constraints at every joint and every contact. It becomes a bookkeeping nightmare.

There is an older, more elegant formulation due to William Rowan Hamilton (1833). Instead of tracking every force, you track a single scalar quantity — the total energy of the system, called the Hamiltonian, denoted $\mathcal{H}$. The remarkable insight: the entire dynamics of any physical system can be derived from this one number.

The Hamiltonian

For any physical system, define the state as $(q, p)$, where $q$ represents generalized positions and $p$ represents generalized momenta.

The total energy $\mathcal{H}$ is conserved: it never changes over time. Once you know $\mathcal{H}$, you get the complete dynamics through two equations:

Concrete Example: The Pendulum

A pendulum has one position variable $\theta$ (angle from vertical) and one momentum variable $p$ (angular momentum). Its Hamiltonian:

If you plot the pendulum's state $(\theta, p)$ over time, it traces a closed curve — the phase portrait. This geometric structure is a direct consequence of energy conservation, and a key diagnostic for verifying that a learned model captures real physics. The separatrix (red dashed curve below) marks the boundary between oscillation and full rotation.

From One Object to Many

A real robotic scene has multiple interacting objects. The Hamiltonian extends naturally. For $N$ objects:

The pairwise interaction term $U_{ij}$ captures every physical interaction between objects — contact, proximity, applied force. This decomposition is the key to compositionality: learn interaction potentials from two-body systems, and they apply automatically to three, four, or ten objects.

3. Learning the Energy Function

In classical physics you write the Hamiltonian analytically. This works for textbook systems but breaks down for the real world. What is the exact Hamiltonian for a robot arm grasping a deformable object on a surface with unknown friction? Nobody can write that equation.

The insight: don't write the Hamiltonian — learn it.

Why Naive Neural Dynamics Fail

A standard neural network trained to predict the next state from the current state has a fundamental flaw. At each timestep it can make a small error — predicting slightly too much or too little energy. Over many timesteps these errors accumulate. The trajectory slowly drifts into physically impossible territory: objects gain energy from nowhere, a pendulum gradually swings higher and higher.

Hamiltonian Neural Networks

Instead of learning the dynamics, learn the energy function. Train a neural network $\mathcal{H}_\theta(q, p)$ that outputs a single scalar — the total energy. Then use automatic differentiation to derive the dynamics:

Training the Firmware

The firmware is pretrained on trajectories from diverse simulated physical systems: single and multi-body rigid dynamics, articulated chains, contact-rich interactions, varied physical parameters (mass, friction, restitution, geometry). The training loss is mean squared error over predicted trajectories:

Trajectory length is annealed during training: start with $k=1$ for stable gradients, gradually increase to $k=50$–$100$ for long-horizon consistency. Without this curriculum the optimization is unstable.

Symplectic Integration

Standard numerical integrators (Euler, RK4) slowly destroy energy conservation through accumulated error — even though the analytical dynamics conserve energy perfectly. The firmware uses the Störmer–Verlet symplectic integrator, which preserves the geometric structure of Hamiltonian systems:

SE(3) Equivariance

Physics does not change if you rotate or translate the entire world. The firmware encodes this via SE(3)-equivariant network layers:

Dissipation: Port-Hamiltonian Extension

Pure Hamiltonian mechanics is conservative — no friction, no energy loss. Real systems dissipate energy. The firmware handles this through the port-Hamiltonian framework, adding a learned dissipation function $R_\psi(p)$ constrained to be positive semi-definite (via Cholesky: $R = LL^\top$):

Object-Centric Graph Structure

4. Integration: A Drop-in Dynamics Module

A robotics company already has a robot, a perception stack, and a control framework. They are not going to rip all of that out. They need something that plugs into what they already have and makes one specific thing better: learning new tasks faster with less demonstration data.

The firmware is a black box that takes in a physical state description and outputs predictions about what will happen next. The interface is simple: their existing perception stack feeds the inputs; their existing planner uses the predictions; their existing controller executes the outputs. You replace one component — the dynamics model — not the whole system.

# Physical Firmware — Python SDK
from physical_firmware import Firmware

fw = Firmware.load("v1.0")     # pretrained, frozen

# Their perception stack provides this:
scene = {
    "objects": [
        {"pos": [0.5, 0.1, 0.8], "vel": [0, 0, 0],
         "mass": 0.3, "shape": "box", "size": [0.05, 0.05, 0.05]},
        {"pos": [0.3, 0.0, 0.9], "vel": [0.1, 0, -0.2],
         "mass": 0.1, "shape": "sphere"},
    ],
    "robot": {"joint_positions": [...], "joint_velocities": [...]},
    "action": {"joint_torques": [0.5, -0.3, 0.1, ...]},
}

# Firmware predicts what happens — single forward pass, ~1ms
prediction = fw.predict(scene, horizon=0.5)
# → predicted positions, velocities of all objects at t+0.5s

# Their planner optimizes over firmware predictions
plan = their_planner.optimize(fw.predict, goal_state, horizon=2.0)

Robot Adapter: One-Time Calibration

The firmware reasons in an abstract physical latent space. A real robot operates in sensor space (cameras, joint encoders, force/torque sensors) and actuator space (joint torques or position commands). The perception encoder $P_\alpha$ bridges this:

When a customer first integrates the firmware, there is a one-time calibration. The robot does ~30 minutes of self-supervised random interaction — pushing objects, picking things up, dropping them. The firmware compares its predictions against what actually happened and fine-tunes a thin adapter layer (not the firmware itself) to account for that robot's specific characteristics: kinematics, gripper friction, sensor noise. Each calibrated adapter is specific to that customer's hardware — this creates lock-in.

Action Interface: MPC or Policy

Two architectures for converting firmware predictions into motor commands:

The Latent Space is Verifiable

The firmware operates in a learned latent space where geometry encodes physical structure. In this space, a pendulum's trajectory traces the phase portrait ellipse not because it was told to, but because the Hamiltonian prior forces that structure to emerge. This is testable: train a linear probe from latent states to known physical quantities (angle, velocity, energy). If $R^2 > 0.95$, the latent space has learned physically meaningful representations — not statistical pattern matching.

5. The Sample Efficiency Claim

The mechanism is precise: the firmware already encodes the physical dynamics of the world. Task-specific training only needs to learn the task structure — not re-derive physics from scratch. This is the same reason a human expert in one sport learns a new sport faster than a non-athlete: the physical foundation transfers.

The definitive validation experiment: select a standard robotic manipulation benchmark (block stacking, pouring, peg insertion). Train three systems — (a) behavioral cloning from scratch, (b) fine-tuning from a behavioral foundation model (RT-2 or equivalent), (c) firmware + policy learning. Plot task success rate vs. number of demonstrations. The firmware system should reach 90% success with 10–50× fewer demonstrations.

The pricing argument follows directly: "You currently need 1,000 demonstrations to teach your robot a new task. Each demonstration costs $X in operator time and robot wear. With our firmware, you need 50 demonstrations. We just saved you 950 × $X per new task."

Comparison with Existing Approaches

The Path