When we describe our calibration system as using reinforcement learning, we usually get one of two reactions. The first is skepticism: isn't this just a PID controller with extra steps? The second is over-extrapolation: does the system learn to design new experiments? Neither reaction is right. This post explains precisely what RL is doing in our calibration stack, why we chose it over simpler alternatives, and where its limits are.
Why classical feedback control is not sufficient
A PID (proportional-integral-derivative) controller is appropriate when the relationship between your control signal and your output is approximately linear, the system has one or at most a few coupled degrees of freedom, and the operating point does not change much over time. For a single-frequency laser lock, a PID controller is exactly the right tool and works extremely well.
An optical tweezer array for neutral-atom quantum computing violates all three assumptions. The relationship between control inputs (AOM drive frequency, trap laser power, beam pointing) and output (gate fidelity) is nonlinear and depends on the current state of all other degrees of freedom simultaneously. The system has on the order of 50-100 independently monitored parameters at any given time. And the operating point changes: atoms in different loading configurations, different circuit depths in user experiments, and seasonal changes in lab conditions all shift the distribution of system states the controller encounters.
More fundamentally, PID control is reactive: it corrects errors after they appear. For quantum hardware, reactive correction has a cost. Running a calibration probe sequence to detect a drift event takes time away from experiments. Running correction pulses after detecting drift risks introducing additional noise from imperfect correction. The ideal calibration system anticipates drift before it degrades performance, adjusting parameters preemptively to stay ahead of it.
Predictive control requires a model of the system's drift dynamics. For our system, those dynamics are not known analytically. The coupling between lab temperature, optical table expansion, AOM driver temperature, and trap frequency involves material thermal expansion coefficients, optical path geometry, and rf thermal response that we do not have precise values for and would be prohibitively expensive to measure systematically. The alternative is to learn the dynamics from data, which is where the ML approach enters.
The RL formulation
We frame the calibration problem as a Markov decision process (MDP). The state is the current observation vector (sensor readings, trap frequency measurements, gate performance metrics). The action is a vector of control setpoint adjustments (AOM frequency offsets, tip-tilt mirror positions, clock laser cavity lock offset). The reward is the negative gate error rate from the most recent benchmarking checkpoint. The agent's goal is to find a policy (a mapping from state to action) that maximizes the expected cumulative reward, i.e., keeps gate error rate low over time.
We use a model-based RL approach rather than a model-free approach. Rather than learning the policy directly from environment interactions using a method like PPO or SAC, we first learn a dynamics model (how the state evolves under different actions) from the historical data collected during months of manual and rule-based operation. We then use this learned dynamics model in a planning loop: at each control step, the agent simulates multiple action sequences forward in time using the learned model and selects the action that leads to the best predicted future reward.
The advantage of the model-based approach for this application is sample efficiency. The safety constraint on our system is strict: we cannot freely explore large action spaces while the quantum processor is running experiments, because large corrections can take the system far from calibrated operating points. A model-free RL agent trained in the live environment would require extensive exploration that is incompatible with providing usable hardware access to research partners. By learning the dynamics model offline from historical data and doing planning rather than direct policy gradient updates, we avoid live exploration while still adapting to new system states.
The dynamics model architecture
The dynamics model predicts the next state given the current state and action. We use a probabilistic neural network that outputs not just a point prediction of the next state but a distribution over next states, parametrized as a mixture of Gaussians. Uncertainty quantification is critical for safe planning: when the model is uncertain about the outcome of an action, the planning algorithm can account for this uncertainty and prefer conservative actions with narrowly bounded predicted outcomes over aggressive actions with high predicted reward but high uncertainty.
The model is trained on transition data from operational history. Each record is a (state, action, next state) tuple collected from the system during both manual calibration sessions and earlier automated operation. The dataset for our current model contains approximately 2.3 million such transitions from 200 days of operation. We retrain the model monthly on the full updated dataset rather than continuously, because continuous online learning introduces the risk of catastrophic forgetting when the current distribution of system states temporarily diverges from historical patterns.
What "learning" means in this context
When we say the system "learns" the calibration dynamics, we mean the dynamics model captures patterns in drift data that are not captured by any rule we have written explicitly. Examples from the learned model that we verified by inspecting the model weights and feature importances: the system learned that trap frequency drift on hot afternoons correlates with magnetometer readings at a specific lab location (near the HVAC duct), allowing it to predict trap frequency drift from the magnetometer 8-12 minutes before the drift is directly measurable. It learned that a specific pattern of cavity photodetector noise is a precursor to a clock laser frequency jump, allowing it to issue a preemptive correction to the cavity lock setpoint before the frequency jump affects qubit coherence. Neither of these correlations was in our manual rule set; both appear as high-weight features in the learned model.
These discoveries are operationally useful. They are also a good illustration of what RL-based calibration does and does not do. The model discovered correlations in the data it was trained on. It did not deduce the physical mechanism behind those correlations; it does not "know" that the HVAC duct drives optical table thermal expansion. It found a statistical relationship between variables in the historical data and uses that relationship for prediction. This is valuable, but it means the model is only as reliable as the data distribution it trained on.
Boundaries and failure modes
We want to be direct about where the current system fails. The planning horizon is limited to approximately 15 minutes. For slow drift events that evolve over hours (such as seasonal temperature excursions), the model's short planning horizon means it is always correcting for the immediate near-term drift rather than anticipating the full trajectory. This is better than no correction but does not fully exploit the predictability of slow drift trends. Extending the planning horizon requires either a longer-horizon dynamics model (which needs more training data to be reliable over longer intervals) or a hierarchical planning approach that we are prototyping but have not deployed.
The system also cannot handle state variables that were not present in the training distribution. If we add a new sensor to the system (a new temperature probe, a new diagnostic output), the model treats it as an unseen variable and ignores it until the dynamics model is retrained with the new sensor included. During hardware changes, we operate with a manually identified subset of controls until enough post-change data accumulates to support a meaningful model update.
Finally, the RL framework is not a substitute for keeping the hardware well-maintained. The calibration system can compensate for slow drift within its actuation range, but it cannot compensate for a degraded fiber coupler, a failing AOM crystal, or a contaminated vacuum window. The reward signal will degrade in response to these events, alerting operators to investigate, but the policy's corrective actions will not be effective because the problem is outside the learned dynamics. The system's value is in eliminating routine calibration labor, not in replacing hardware maintenance or expert troubleshooting.