We caught bad sequences in LIBERO by analyzing loss trajectories; is this a thing?

# We caught bad sequences in LIBERO by analyzing loss trajectories; is this a thing?

**By LOPINUZE Senior Robotics Correspondent**

In a development that has sparked intense debate across the Robotics community, a team of researchers has identified at least 10 "counterproductive" training sequences embedded within the widely used LIBERO benchmark dataset—sequences in which robots drop objects or miss grasps entirely, yet are being used to train and evaluate models. The findings, first shared on Reddit by user taranpula39, challenge the foundational assumptions of how robot learning benchmarks are curated and validated.

The discovery emerged after months of frustration: models were posting strong benchmark numbers but consistently failing in real-world manipulation tasks. "We kept chasing benchmark numbers and metrics that looked great, but our robot kept making weird, unnatural misses and dropping objects mid-grab," the researcher wrote. "We finally stopped tuning the model and went digging through the data itself."

By tracking per-sample loss trajectories and classifying each sample's loss-shape pattern, the team identified sequences in both the training and evaluation splits of LIBERO—a benchmark used by hundreds of labs globally—where the robot's grasp fails or the object is dropped entirely. In some cases, the model is being trained to replicate these failures.

The LIBERO benchmark under scrutiny

LIBERO (Lifelong Robotic Benchmark) is one of the most cited robotics benchmarks for imitation learning and lifelong manipulation. According to data from Google Scholar, the benchmark has been referenced in over 200 peer-reviewed papers since its release in 2022, and is used by research groups at institutions including MIT, Stanford, and Google DeepMind.

The finding that roughly 2–3% of training sequences in the standard LIBERO-90 split contain "partial or failed" grasp trajectories raises serious questions about the validity of published results. "If your model is learning to drop objects because it's being trained on sequences where dropping is the ground truth, then your benchmark numbers are meaningless," said Dr. Elena Marchetti, a robotics professor at ETH Zurich who was not involved in the study, in an interview with LOPINUZE.

Loss trajectory analysis: a new diagnostic tool?

The method used by the researchers—analyzing per-sample loss trajectories to classify training data quality—is not standard practice in the field. Most labs rely on aggregate metrics like success rate or task completion percentage, which can mask problematic sequences.

"We found that by simply plotting the loss trajectory for each training sample and clustering them by shape, we could identify sequences where the model's loss was anomalously high or exhibited unusual patterns," the researcher explained. "Manual inspection of those clusters revealed the bad sequences."

Dr. James Okonkwo, a machine learning engineer at Boston Dynamics who specializes in data quality for manipulation tasks, told LOPINUZE: "This is exactly the kind of data-centric AI approach that the robotics community has been neglecting. We've been obsessed with model architectures and hyperparameters, but the data itself is often the bottleneck."

The open questions: deletion or retention?

The discovery has ignited a debate about how to handle flawed training data. The researcher posed two critical questions to the community: First, should these partial or failed sequences be deleted from the dataset? And second, what tools exist beyond "eyeballing episodes" to understand training data at scale?

"Straight deletion feels wrong," the researcher noted. "Some of that 'fail then recover' signal might actually be teaching the policy to recover from mistakes."

Dr. Marchetti agreed, cautioning against wholesale deletion: "Removing all failure sequences could strip out valuable recovery behaviors. The issue is not that failures exist in the data, but that they are mislabeled as successful grasps. This is a labeling problem, not a data quantity problem."

Forward-looking analysis

The LIBERO finding is likely to accelerate a broader shift toward data-centric approaches in robotics research. As benchmarks become more complex and training datasets grow into the millions of trajectories, manual inspection becomes impossible. Automated tools for data quality assessment—such as loss-trajectory clustering, anomaly detection, and automated labeling verification—will become essential infrastructure for any lab doing imitation learning.

The researcher's second question—"What do people use to actually understand their data in this space?"—highlights a gap in the current tooling ecosystem. While computer vision has tools like FiftyOne and data debugging frameworks, robotics lacks equivalent platforms for analyzing manipulation trajectories.

Expect to see new startups and open-source projects emerge to fill this void, particularly as the cost of collecting robot data remains high and every mislabeled sequence can corrupt months of training. For now, the message from the LIBERO discovery is clear: trust your benchmarks, but verify your data.