Latent-variable policies
An encoder maps demonstrated actions to a latent code z, and a decoder models p(a | s, z). During deployment, z is sampled from a prior. CVAE-based policies must retain action-mode information while matching this prior.
Understanding Multimodality in Generative Behavioral Cloning
NCT/UCC Dresden · TU Dresden · DKFZ · LMU Munich · MCML
Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026)International Convention Centre Sydney · Sydney, Australia
Abstract
Behavioral cloning becomes challenging when the same observation admits several valid actions. We study how generative behavioral-cloning policies represent such multimodal expert behavior and identify different bottlenecks across model parameterizations. For latent-variable policies, preserving demonstrated modes requires action-conditioned information in the latent representation. Excessive posterior–prior regularization can suppress this information and prevent the policy from distinguishing demonstrated modes. Weaker or aggregate regularization can preserve mode information, but shifts the challenge to ensuring that the deployment-time prior covers the relevant latent regions. For action-space generative policies, multimodality is constrained by the smoothness of the base-to-action transport: a map with a small Lipschitz constant cannot assign substantial probability to many well-separated modes. Covering many modes therefore requires either sharp transitions in base space or off-support bridge regions in action space. Experiments on synthetic multimodal navigation and a physical-robot bimodal manipulation task support these mechanisms. In contrast, our analysis reveals limited conditional multimodality in standard robotic simulation benchmarks, where deterministic regression remains competitive.
Background
We study two classes of generative behavioral-cloning policies. Both predict an action chunk conditioned on the observation, but they represent action ambiguity differently.
Action-space generative policies
Diffusion and flow policies transform base noise into actions, conditioned on the observation. Their ability to represent separated modes depends on the geometry and smoothness of this base-to-action map.
Main Results
Latent-variable policies
Latent-variable policies require action-conditioned information in the latent code to distinguish demonstrated modes. Excessive posterior–prior regularization suppresses this information. As the latent becomes uninformative, the decoder loses the information needed to distinguish the actions and predicts their average.
Latent posterior q(z | a, s)
a = −1a = +1Prior p(z)
Action distribution p(a | s, z)
p(a | s, z₁)p(a | s, z₂)Demonstrated actions
Mode information— bits
Action-space generative policies
For diffusion and flow policies, separated action modes must be represented by a map from base noise to action chunks. A map with a small Lipschitz constant cannot place substantial probability on many well-separated modes without assigning probability to invalid actions between them.
Base-to-action map G(u)
Inputs producing invalid actions
0.58
Invalid bridge probability—
Synthetic navigation experiments
We evaluate four tasks with known action modes. Policies receive a single RGB observation and predict a 60-step action chunk without replanning.
Circle (2 modes)
Sequential (4 modes)
Radial (16 modes)
Corridor (16 modes)
Deterministic regression averages the demonstrated modes and achieves 0% success on all four tasks. Generative policies recover high mode coverage, but on the two 16-mode tasks success ranges from 37% to 52% despite near-complete coverage. Mode coverage alone does not ensure valid trajectories.
KL regularization and latent mode information
Lipschitz bounds and invalid bridges
Robotic manipulation
Simulation datasets
Bimodal tissue grasping
We evaluate a surgical robot on a tissue-grasping task with two demonstrated approaches. Deterministic regression averages the modes and achieves 0% success. CWAE (Sinkhorn, learned prior) collapses to one mode while retaining 85% success, illustrating that successful task execution does not imply recovery of the demonstrated distribution.
Conditional multimodality in demonstration datasets
Standard robotic benchmarks exhibit very little multimodality, while the proof-of-concept tissue-grasping task exhibits two distinct action modes.
On the nearly unimodal simulation datasets, deterministic regression remains competitive. Generative policies can improve or worsen task performance, depending on their training dynamics.
Estimated conditional mode count
Action horizon: 30 steps
Policy performance
Average inference speed
Conclusion
- Deterministic regression performs reasonably well across the simulation benchmarks, consistent with limited conditional action multimodality in their demonstration data. It also achieves higher average inference throughput than the flow-matching and CVAE policies evaluated here. The choice of a multimodal-capable policy should therefore be informed by whether the demonstration data require this representational capacity.
- The performance of multimodal-capable policies varies with training dynamics. Latent-variable policies involve many hyperparameters, resulting in a large search space for an effective configuration. Action-space flow or diffusion policies may therefore be preferable when reducing this tuning burden is a priority.
- Some models collapse late in training, including flow matching on UR3 BlockPush and CVAE (KL, learned prior) on Kitchen Image. These observations highlight the importance of checkpoint selection and motivate further investigation of the role of covariate shift in performance degradation.
- The bimodal tissue-grasping task illustrates the consequences of failing to represent certified multimodal demonstrations. Deterministic regression collapses toward an average between modes, whereas CWAE (Sinkhorn, learned prior) collapses to a single demonstrated mode. In this case, single-mode collapse retains a demonstrated strategy rather than interpolating into a region outside the expert support. Although CWAE (Sinkhorn, learned prior) does not recover the full multimodal behavior, it still achieves an 85% success rate.
BibTeX
@article{mazza2026understanding,
title={Understanding Multimodal Failure in Action-Chunking Behavioral Cloning},
author={Mazza, Lorenzo and Datres, Massimiliano and Rodriguez, Ariel and Bodenstedt, Sebastian and Kutyniok, Gitta and Speidel, Stefanie},
journal={arXiv preprint arXiv:2605.22493},
year={2026}
}