voxel51

How to Find Edited Copies of Images: Image Copy Detection with pHash, CLIP, and FiftyOne Exact-duplicate search misses a mirrored, cropped, or filtered copy of an image, so this post combines a perceptual hash and a CLIP embedding into one rule you control and shows the copies as a graph in FiftyOne. Your model's test metrics look great. Then someone opens the test set and finds a mirrored copy of a training image. Then another one, with a filter on it. Then a third, cropped and covered in emoji. You run a duplicate check. It finds nothing. That's because duplicate checks look for the same bytes, and an edited copy has different bytes. A mirrored JPEG is a different file from the original. To a hash of the file contents, the two are strangers. To a person, they're the same picture. The task is called image copy detection. Given a query image, decide whether it's an edited copy of an image in a reference collection, and say which one. This post walks through one practical approach: compute two signals per image, combine them with a rule you control, and read the result as a graph where every link comes with evidence. It's packaged as an open-source FiftyOne plugin called Image Link Lab, but the idea carries over to any dataset where copies hide behind edits. !Image Link Lab in FiftyOne. Tune the pHash and CLIP limits, and the copy graph redraws as you drag. Image Link Lab in FiftyOne. Tune the pHash and CLIP limits, and the copy graph redraws as you drag. Key takeaways - Image copy detection asks whether a query image is an edited copy of a reference image, and which one. compute_exact_duplicates can't answer that, because edits change the bytes. - A perceptual hash (pHash) summarizes pixel layout in 64 bits. It tolerates recompression and small blurs and breaks on mirrors, rotations, and big crops. - A CLIP embedding (clip-vit-base32-torch from the FiftyOne model zoo) summarizes what the picture shows. It survives mirrors and crops and gets fooled by look-alikes. - Each signal's blind spot is the other's strength, so the plugin computes both and lets you choose how they combine: link a pair if either signal passes (combine: "any"), or only if both do ("all"). - Moving a limit changes the answer. The Image Link Lab panel redraws the links as you drag the sliders and shows the numbers behind every line. Given a truth field, it scores each rule with FiftyOne's native evaluation so you can compare rules in Model Evaluation. - The output is ordinary FiftyOne data: a phash field, a copy_of Classification field, and copies_* evaluation runs. You can tag copies, save views, and filter a test split cleanly with the code you already use. What is image copy detection? Image copy detection is the task of deciding whether a query image is a modified copy of any image in a reference corpus. The edits can be mundane (crop, resize, or re-compress) or hostile (mirror, overlay text, blend with another image, or screenshot it on a different device). The standard benchmark is DISC21, the dataset behind the Image Similarity Challenge at NeurIPS 2021. Its authors describe the goal as determining "whether a query image is a modified copy of any image in a reference corpus of size 1 million," with edits that include automated transformations, hand-crafted changes, and machine-learning-based manipulations. Both the query set and the reference set are mostly distractors that match nothing, a needle-in-a-haystack setting like real life. The paper is The 2021 Image Similarity Dataset and Challenge. Copy detection is a different question from "which images look like this one?" A similarity search returns neighbors. Copy detection has to commit: this query is a copy of that original, or it isn't a copy of anything. !One beach photo and five edited copies (mirrored, cropped, blurred, text overlay, and grayscale with rotation) stored on five different devices. Exact-duplicate search finds none of them. A side panel lists why finding them matters: train/test leakage, labeling cost, licensing, and forensics. Five edited copies of one image are five different files, so exact-duplicate search finds none of them. Where edited copies cost you You don't need a content moderation team to care about this. Four situations come up again and again: - Train/test leakage. A mirrored copy of a training image in your test set inflates your metrics. You don't find out until the model meets real data. - Labeling cost. You pay to label the same picture several times, sometimes with conflicting answers. - Licensing and attribution. You need to know where an image came from, and a cropped copy hides its source. - Moderation and forensics. A known-bad image comes back filtered, or shows up on a second device. The first two hit any training pipeline. The last two are what DISC21 was built around: its authors say the benchmark mimics social media problems such as misinformation and objectionable content. Why exact duplicates and single embeddings fall short FiftyOne already handles two nearby problems. compute_exact_duplicates finds byte-identical files. compute_near_duplicates, along with the Similarity Search panel, finds neighbors by a single embedding. Neither is built for edited copies. Exact matching fails the moment a pixel changes. A single embedding gives you a ranked list of look-alikes, with one similarity score per pair and no way to say which kind of evidence you trust. If the score is wrong, you can't tell whether the embedding was fooled or the copy was over-edited. What's missing is a way to combine two different kinds of similarity with a rule you control, and a way to see which image is linked to which, and why. Two signals, two blind spots The approach starts from a simple observation. Pixels and meaning fail on different edits. A perceptual hash (pHash) summarizes an image's pixel layout in 64 bits. Two images whose hashes differ in only a few bits look the same at a glance. It's fast, needs no model, and shrugs off recompression and a small blur. It's defeated by anything that moves pixels around: a mirror, a rotation, or a big crop. A CLIP embedding summarizes what the picture shows. Two images with a high cosine similarity are about the same thing. It survives mirrors and crops. It's fooled by look-alikes: two different photos of the same kind of scene can score higher than a heavily blurred copy of the same photo. !Four DISC21 image pairs scored by pHash and CLIP: a mirrored copy only CLIP catches, a striped and blurred copy only pHash catches, a look-alike CLIP nearly links by mistake, and a flipped, blurred, and filtered copy both signals miss. Pixels and meaning fail on different edits, so pHash and CLIP each catch copies the other misses. The four pairs above show it. A mirrored copy is invisible to pHash and obvious to CLIP. A striped and blurred copy is the reverse. A look-alike that isn't a copy nearly gets linked by CLIP. And a flipped, blurred, filtered copy beats both. Good to know. Why not train a copy-detection model? You can, and the DISC21 challenge exists to push that line of work. The two-signal approach is for the common case where you have a dataset on disk, no labeled copy pairs, and a question to answer this afternoon. Both signals work off the shelf. Combine them with a rule you control Each signal gets a limit. For pHash, it's the most bits two hashes may differ by. For CLIP, it's the least cosine similarity two images must reach. Then you choose how the signals combine: - Either signal is enough (any): link the pair if pHash or CLIP passes. You get more links, and more wrong ones. - Both must agree (all): link only if both pass. You get fewer links, and almost all of them right. Moving a limit is the whole game. Loosen the pHash limit, and you catch more heavily edited copies and start linking images that only look alike. Tighten it to all, and the wrong links disappear along with most of the right ones. No setting gets everything. That's why the plugin doesn't give you a single "is copy" score. It shows you where the trade-off sits for your data and lets you pick. Keep it honest. There is no universal threshold. A limit that works on re-uploaded product photos will behave differently on a collection full of heavy crops and overlays. If you have even a small set of known copy pairs, use it to calibrate the rule. Where FiftyOne and its plugin ecosystem fit FiftyOne is an open-source tool for building high-quality datasets and visual AI models. It gives you the dataset layer: samples with fields you can filter and sort on, views, tags, the App for looking at your images, and native tools for similarity search and model evaluation. It can't ship every check that depends on your data. That's the job of plugins. A FiftyOne plugin is a folder FiftyOne discovers at runtime. It includes a manifest (fiftyone.yml), Python code, and optionally a React front end. You add a button, a form, a background job, or a whole panel to the App without forking FiftyOne or waiting for a release. Plugins come in two kinds: - Operators are actions. You trigger one, it runs, it finishes. - Panels are UI that stays open next to your data and reacts when you filter the grid. Plugin code runs on the FiftyOne server in Python so that it can use NumPy, PyTorch, a GPU, or any library you can pip install. Long jobs can be delegated to a background worker so the App stays free. Every operator also has a URI, so anything that can call it, from a button to a notebook to an AI agent, runs the same code. For a deeper tour of how operators and panels work, see Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins. Image Link Lab is built from standard parts: two operators (compute_signals, find_copies), one hybrid panel (image_link_lab), native similarity indexes, and native evaluation runs. The plugin adds only what FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph. !The five-step Image Link Lab workflow: compute signals, find copies, open the panel, score the rule, and act. Each step lists what is plugin code and what is native FiftyOne. The plugin adds three things FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph. Everything else is native. Step 1: Install the plugin and compute signals Install the plugin and its requirements: fiftyone plugins download https://github.com/harpreetsahota204/image_link_lab fiftyone plugins requirements @harpreetsahota/image_link_lab --install The plugin needs fiftyone>=1.22.1. CLIP runs on CPU but is much faster on a GPU. Open any image dataset in the App, click + next to Samples, and add the Image Link Lab panel under Custom. Click Compute signals, keep both boxes ticked, and execute. !Computing the pHash and CLIP signals with the Image Link Lab operator. Computing the pHash and CLIP signals with the Image Link Lab operator. The operator writes three things: | Where | What | | --- | --- | | phash field | the 64-bit pHash of each image, as hex | | brain key phash | the hash bits as a native similarity index, so sort-by-similarity in the App works on pHash too | | brain key clip | a native CLIP similarity index (clip-vit-base32-torch from the model zoo) | What the compute signals operator writes to your dataset: a phash field and two native similarity indexes. It runs delegated by default, so start a worker with fiftyone delegated launch or choose "Execute now" in the form. pHash takes about a minute per 10,000 images, plus CLIP time. Signals that are already computed are left alone unless you tick them. Step 2: Find copies Click Find copies and fill in the form. !Filling in the find copies form: queries, candidate originals, an optional truth field, and the starting rule. Filling in the find copies form: queries, candidate originals, an optional truth field, and the starting rule. - Query images are the ones you want explained. Each is linked to at most one original. - Candidate originals are where to look. Both can be the current view, the whole dataset, or a tag. They can overlap, and an image is never linked to itself. - ID field (optional) is what to call each image, if your dataset identifies images by something other than the sample ID. - Truth field (optional) holds, for each query, the ID of its known original. This is your ground truth, supplied by your dataset, not computed by the plugin. Set it, and you get scoring. Leave it empty, and the plugin works unsupervised: links are drawn, nothing is judged. - Rule sets the starting pHash limit, CLIP limit, and any or all. You'll change these in the panel, so this is just where the first result comes from. Behind the form, the operator does three things: 1. Gathers candidates. For each query, it takes the 5 nearest originals by pHash and the 5 nearest by CLIP, plus the true original if a truth field is set. It stores both signals' values for every candidate. This is the slow step, about 4 seconds for 1,000 queries against 5,000 originals, and every later run reuses it. 1. Applies the rule and writes copy_of, a Classification per query whose label is the linked original's ID, or none. copy_of is a prediction from your rule, not a human label. When several candidates pass, the one passing the most signals wins, then the highest CLIP similarity, then the fewest differing bits. Scores, if a truth field is set. !The find copies form beside the four steps its execute function performs: gather candidates once, apply the rule, score it with a native binary evaluation, and remember the settings for the panel. The slow step, gathering candidates, runs once. Every rule you try after that reuses them. Step 3: Read the graph The panel draws whatever queries are in the current grid view, up to 40 at a time. Filter the grid and the graph redraws. - Nodes are images. Copies have a gray border. Originals have an orange border and, when more than one copy points at them, a badge with the count. - Lines are candidate pairs the rule has something to say about. With a truth field set, green is a correct link, red is an incorrect link, and dashed amber is a missed original. Thick lines pass both signals, and thin lines pass one. - Images joined by lines form families, laid out as a star with the original in the middle. A loose rule joins families through wrong links, and you'll see red lines bridging two stars. - Queries with nothing to draw sit in a strip at the bottom. For a distractor with no original, that's the correct outcome. Click an image and the grid filters to that image and everything it has a line to, including originals your current view doesn't contain. Click a line and an evidence pane opens with the copy and the original side by side, then one row per signal: pHash pixels 30 bits differ limit ≤ 10 fails CLIP meaning 0.731 similar limit ≥ 0.90 fails Below the rows is a verdict in a sentence, such as "Not linked: pHash and CLIP are outside the limit. This is the true original, so the rule missed it." You can check every statement against the numbers above it. No model decides behind the scenes. Move the sliders. The rule runs in the browser against the stored candidate values, so lines and counts update as you drag, with no server round trip. Good to know. Does the graph work without ground truth? Yes. Without a truth field, every link is drawn green, and nothing is dashed. The colors mean "linked," not "right." Step 4: Score the rule and compare alternatives If you set a truth field, click Score this rule. The plugin runs the rule over every query, not just the 40 on screen, writes copy_of, and records a native FiftyOne evaluation named after the rule, such as copies_phash10_or_clip90. Then it opens Model Evaluation. The evaluation is binary, per query. Ground truth is copy if the query has a known original and unique if it doesn't. The prediction is copy if the rule linked the query to its true original, and unique otherwise. FiftyOne computes precision, recall, F1, and a 2×2 confusion matrix from those two labels. One definition matters: a copy linked to the wrong original counts as a miss, not a false positive. The evaluation asks whether you found the true original, and the answer is no. The loop looks like this: 1. Score the starting rule and open its evaluation in Model Evaluation. 1. Click the false-negative cell. The grid filters to the misses, and the graph redraws them with dashed amber lines to their originals. Click a line to see which signal failed and by how much. That tells you which slider might help. 1. Change a limit, score again, and you get a second evaluation key. 1. Open one and compare it with the other. 1. Mark the rule you keep as Reviewed and delete the rest with dataset.delete_evaluation(key). The false-positive cell is the other one worth clicking. Those are distractors the rule linked, usually look-alikes, which is CLIP's failure mode. Comparing a pHash-only run with a CLIP-only run shows which edits each signal survives. Keep it honest. Model Evaluation scenarios, which break results down by edit type, are a Voxel51 feature. In open-source FiftyOne, filter the grid by an edit-type field and read the chips, or compute the numbers in Python. Step 5: Act on the results copy_of is an ordinary label field, so everything else in FiftyOne works on it: from fiftyone import ViewField as F copies = dataset.match(F("copy_of.label") != "none") copies.tag_samples("copy") dataset.save_view("copies", copies) # Keep a test split clean clean_test = dataset.match_tags("test").match(F("copy_of.label") == "none") # Everything linked to one original dataset.match(F("copy_of.label") == "R003513") Typical setups differ only in what you pass as queries and originals: | Situation | Queries | Originals | Truth | | --- | --- | --- | --- | | Dedup a collection | whole dataset | whole dataset | none | | Check a test split for leakage | tag test | tag train | none | | Match seized images against a known-bad list | the seized images | the known-bad list | none, or a case label | | Benchmark a rule | the edited images | the originals | the original's ID | Typical Image Link Lab setups, with what to pass as queries, originals, and truth for each. Run the same workflow from Python Everything the App does is an operator, so a notebook, a script, or an agent can run it: import fiftyone as fo import fiftyone.operators as foo dataset = fo.load_dataset("disc21_link_lab") # Step 1: signals (delegate=False runs it in this process) foo.execute_operator( "@harpreetsahota/image_link_lab/compute_signals", {"dataset": dataset, "params": {"phash": True, "clip": True, "delegate": False}}, ) # Step 2: copies, with a rule, scored against a truth field result = foo.execute_operator( "@harpreetsahota/image_link_lab/find_copies", { "dataset": dataset, "params": { "queries": "tag:final_query", # or "view", "dataset", "tag:<tag>" "originals": "tag:reference", "id_field": "disc21_id", # optional "truth_field": "reference_id", # optional; turns on scoring "phash_max": 10, "clip_min": 0.90, "combine": "any", # "any" or "all" "delegate": False, }, }, ) print(result.result) # The results are ordinary FiftyOne fields and runs dataset.load_evaluation_results("copies_phash10_or_clip90").print_report() Run find_copies again with a different rule and the candidates are reused, so it takes about a second and adds another evaluation run to compare. !One operator URI per step, callable from panel buttons, the find copies form, a notebook, an AI agent, or a background worker. Native tools close the loop between the grid, the Image Link Lab panel, scoring, and Model Evaluation. Every step is an operator with one URI, so the App, a notebook, and an AI agent all run the same code. Both operators have typed inputs and show up through the FiftyOne Model Context Protocol (MCP) server (list_operators, get_operator_schema, execute_operator), so an AI agent can compute signals, try a rule, and read back precision and recall. How it's built The plugin is small on purpose. The Python operators and the pure-Python core (hashing, candidate gathering, the rule) are tested with pytest. The panel UI is built in React with VOODO, the component library the FiftyOne App itself uses. The rule exists twice on purpose: in Python for the operators and in TypeScript so the sliders can run it in the browser. A shared fixture, tests/fixtures/rule_cases.json, feeds both test suites, so the two can't drift. !The image_link_lab plugin folder and its four layers (browser, framework, engine, and core), each calling only the layer below it, with the rule mirrored between rule.py and rule.ts. Four layers, each calling only the one below. A shared test fixture keeps the Python and TypeScript rules in sync. Everything else is native FiftyOne. CLIP comes from compute_similarity, the pHash bits are registered as a second similarity index, results are plain Classification fields, scoring is evaluate_classifications, and rule history lives in evaluation runs. Keep it honest. The demo dataset, disc21_link_lab, is a subset of DISC21: 1,000 edited queries and 5,000 originals, with a known original for 500 of the queries. The rest are distractors. The full benchmark has a reference corpus of 1 million images, so don't read results from this subset as benchmark numbers. DISC21's edits are also deliberately brutal (heavy crops, overlays, and blends), so recall will be low on it. On a collection of ordinary re-uploads, pHash alone will catch most copies. Try it yourself - Image Link Lab plugin on GitHub - Try the plugin on the DISC21 dataset on this Hugging Face space - FiftyOne on GitHub - FiftyOne plugin docs - The 2021 Image Similarity Dataset and Challenge (paper) - Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins Frequently asked questions What is image copy detection? Image copy detection is the task of deciding whether a query image is an edited copy of an image in a reference collection, and identifying which one. It differs from similarity search, which returns neighbors without committing to a match. How do I find edited or modified copies of an image in a dataset? Compute more than one kind of similarity, since no single signal survives every edit. Image Link Lab computes a perceptual hash in the phash field and a CLIP similarity index, then links pairs with a rule over both. find_copies writes the result to a copy_of field. Can perceptual hashing detect mirrored or cropped images? Not reliably. A pHash summarizes how pixels are laid out, so anything that moves pixels around (a mirror, a rotation, or a big crop) defeats it. A CLIP embedding survives those edits, which is why the plugin uses both. What's the difference between exact duplicates, near-duplicates, and copy detection in FiftyOne? compute_exact_duplicates finds byte-identical files. compute_near_duplicates finds neighbors by a single embedding. Image Link Lab links each query to at most one original using two signals and a rule you set, and shows the evidence for every link. How do I check a test split for train/test leakage? Run find_copies with queries set to tag:test and originals set to tag:train. Then match copy_of.label != "none" on the test split to see which test images link to a training image. Do I need ground truth labels? No. Without a truth_field, the plugin draws links, and nothing is judged. With one, you get right, wrong, and missed colors, counts, and a native evaluation per rule. How large a dataset can it handle? Candidates are gathered with dense distance matrices, which works fine for tens of thousands of originals. Beyond that, narrow the originals to a tag or a view. Can I use a different embedding than CLIP? Not from the form. The plugin is deliberately opinionated: one signal for pixels, one for meaning. To swap models, change CLIP_MODEL in constants.py to any zoo model that produces embeddings. Is CLIP slow? It's the only model in the pipeline, and it runs once. On CPU, budget about a second per image. On a GPU, a few minutes for several thousand.

Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins How the FiftyOne plugin framework turns your MCAP and LeRobot curation checks into buttons, panels, and background jobs that run on every episode, and why that works across a whole dataset rather than one recording. You have 400 teleoperation episodes and a policy to train by Friday. Some of those episodes are bad. The operator hesitated for eight seconds before the grasp. The wrist camera froze halfway through. The gripper opened and closed three times because the first grasp slipped. The task string says "pick up" and nothing else. A policy will learn from all of it. You could open each episode in a log viewer and scrub through it. At two minutes per episode, that's more than 13 hours, and you'll stop paying attention long before the end. What you want is a ranked list, worst first, with every flag one click away from the moment it happened. No viewer ships that list, because the checks depend on your robot. A dual-arm rig with multi-joint hands needs different smoothness metrics than a wheeled rover with an inertial measurement unit (IMU) and GPS. The tool that curates your data has to be one you can extend in Python, on your schema, across the whole dataset. That's what the FiftyOne plugin framework is for. Key takeaways - A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job. - FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code. - FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution. - Most viewer extensions run in the browser against one recording, and labeling-platform hooks act on one workflow task. FiftyOne plugins run server-side Python with custom UI across a whole dataset. - Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings. Why physical AI data needs custom tooling Image datasets are forgiving. Every sample is a picture, every label is a box or a mask, and the same quality checks work on most of them. Robot data has no such standard shape. One MCAP file from an autonomous vehicle might hold six cameras, two LIDARs, an IMU, GPS fixes, and a /tf tree. One LeRobot episode from a tabletop arm holds two camera streams, a 14-dimensional observation.state, a matching action array, and a task string. An egocentric dataset adds audio and hand poses. The questions you need answered are specific to that shape: - Is the motion in this episode smooth, or full of corrections? - Did any sensor drop frames, drift its clock, or clip its values? - Does the action track the state, or is there lag between video and action? - Is the gripper behavior consistent with the other episodes of the same task? - Is the language instruction usable for fine-tuning a vision-language-action (VLA) model? A generic viewer can't answer those. It can only show you the data and wait for you to notice. FiftyOne gives you the dataset layer: episodes as samples, fields you can filter and sort on, views, tags, and an App with a multimodal viewer that plays MCAP and LeRobot streams in lockstep. Plugins give you the part that's specific to your robot. !Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation. Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation. What is a FiftyOne plugin? A plugin is a folder FiftyOne finds. If the folder contains a fiftyone.yml, FiftyOne loads it at runtime and adds whatever it declares to the App, the SDK, and the CLI. Plugins add two kinds of things: - Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator. - Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel. Under the hood, a panel is a special kind of operator, so everything below about operators applies to panels too. | | Notebook script | FiftyOne plugin operator | | --- | --- | --- | | Who can run it | You, in your kernel | Anyone with the App open, no code | | Inputs | Variables you edit by hand | A typed form, validated before anything runs | | What it acts on | The dataset you loaded | The view or selection the user has open | | Long jobs | Your kernel is busy until it finishes | Delegated to a worker, so the App stays free | | Reuse | Copy and paste | One URI: buttons, panels, pipelines, notebooks, agents | What changes when you promote a notebook script to a FiftyOne plugin operator. Good to know. Why not keep it in a notebook? Start in a notebook. Promote the code to an operator when someone else needs to run it, or when you catch yourself running it a second time. The figure below shows what changes. !Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse. Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse. The row that matters most for robot data is "what it acts on". A script scores whatever dataset you loaded. An operator scores the view the user has open right now, so the same code works on one task, one robot, one week of collection, or the episodes someone tagged for review. The anatomy of a FiftyOne plugin Here's the folder for a plugin with a Python backend and a custom React panel: !FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App. FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App. - fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs. - __init__.py holds your classes and a register(p) function. - requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends. - Everything below the dashed line is optional JavaScript for custom UI. A Python-only plugin is two files. Teal in every figure in this post means the FiftyOne server, which runs Python. Orange means the App in your browser. Keep it honest. Every operator name listed in fiftyone.yml must match the name in that operator's config. If you register a class but forget to list it, the operator doesn't show up, and you get no error. It's the most common "where did my operator go" bug. A complete operator in 25 lines This operator tags samples whose detections fall below a confidence threshold: import fiftyone as fo import fiftyone.operators as foo import fiftyone.operators.types as types from fiftyone import ViewField as F class TagLowConf(foo.Operator): @property def config(self): return foo.OperatorConfig(name="tag_low_conf", label="Tag low-confidence") def resolve_placement(self, ctx): return types.Placement( types.Places.SAMPLES_GRID_SECONDARY_ACTIONS, types.Button(label="Tag low-conf"), ) def resolve_input(self, ctx): inputs = types.Object() fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections)) inputs.enum("field", fields, required=True, label="Label field") inputs.float("thresh", default=0.3, label="Confidence threshold") return types.Property(inputs) def execute(self, ctx): field, thresh = ctx.params["field"], ctx.params["thresh"] view = ctx.target_view().filter_labels(field, F("confidence") < thresh) view.tag_samples("low_conf") ctx.ops.notify(f"Tagged {len(view)} samples") def register(p): p.register(TagLowConf) And here's what the user sees: !TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App. TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App. Four methods, four jobs: 1. config names the operator. name is its ID, and label becomes the form's heading. 1. resolve_placement puts a "Tag low-conf" button in the grid's action bar. 1. resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines. 1. execute does the work on the target view and sends a notification. You wrote no UI code. You described the inputs and the App built the interface. Swap the body of execute for spectral arc length (SPARC) smoothness over an /odom channel, and you have the skeleton of an MCAP quality scorer. How an operator call works Every operator call is two round trips between the App and the FiftyOne server: !Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations. Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations. 1. Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel. 1. Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them. Your code runs only on the server. The browser draws what the server tells it to. That's why an operator can decode protobuf MCAP messages with mcap-protobuf-support, run SciPy's Welch power spectral density (PSD) on an IMU stream, or call a model on a GPU, without shipping any of that to the browser. When something breaks, ask which arrow failed. A wrong form means resolve_input. A fine form that does nothing means execute. An execute that ran but didn't change the App means the UI operations you returned. Long jobs run in the background with delegated execution Scoring 400 episodes is not a click-and-wait job. Decoding video alone costs lerobot_data_curation about 0.4 seconds per camera per episode. !Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them. Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them. Set allow_delegated_execution=True in the operator's config, and the user can choose. Immediate execution runs inside the server while the App waits, with a live progress bar. Delegated execution writes the job to a queue in MongoDB, and a separate worker runs the same execute method. You start a worker with fiftyone delegated launch and track jobs with fiftyone delegated list. Both physical AI plugins below default to delegated execution, because a real corpus takes minutes. Keep it honest. If no worker is running, delegated jobs sit in the queue forever. And a delegated job has no browser attached, so write results to the dataset rather than trying to change the user's view. Panels: UI that stays open A curation tool needs more than a form. It needs a place to see the ranking, click into an episode, and tag it. That's a panel. !FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields. FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields. A panel's on_load runs when it opens, render returns a layout, and events call your Python handlers with a fresh context. The event that matters most for curation is on_change_view: when the user filters the grid, your panel re-computes for the new view. Here's a complete panel that shows class counts and follows the grid: class ClassCounts(foo.Panel): @property def config(self): return foo.PanelConfig(name="class_counts", label="Class counts") def on_load(self, ctx): self.refresh(ctx) def on_change_view(self, ctx): self.refresh(ctx) def refresh(self, ctx): counts = ctx.view.count_values("ground_truth.detections.label") ctx.panel.set_state("n", len(counts)) ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}]) def render(self, ctx): panel = types.Object() panel.md(f"**{ctx.panel.get_state('n')} classes**") panel.plot("counts") return types.Property(panel) Small values go in panel state, which your Python handlers can read. Large payloads like chart data go through set_data, which is write-only from Python. Settings you want to survive a reload go in the execution store, ctx.store. Results that belong to the data go in dataset fields, where you can filter and sort on them like any other field. When the built-in widgets run out, you have two options. A hybrid panel keeps state and compute in Python and renders a React component you write. A JS panel puts React in charge of the UI and calls your Python operators whenever it needs the server. Both plugins below are JS panels backed by Python operators, which suits dense, interactive dashboards. If you build a React UI, VOODO, the component library the FiftyOne App itself uses, makes it look native. !Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build. Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build. Operators compose Every operator has a URI, like @voxel51/brain/compute_similarity, and a typed input schema. Anything that can call an operator can call yours. !One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates. One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates. That includes the FiftyOne Model Context Protocol (MCP) server, which exposes list_operators, get_operator_schema, and execute_operator to AI agents. An agent can find your quality scorer, read its inputs, run it on the episodes tagged review, and report back which ones failed sensor health. The schema validates the agent's JSON the same way it validates a person filling in the form. For a physical AI team, that means the curation logic you write once runs from a button during review, from a notebook in your training pipeline, from a background worker overnight, and from an agent. How FiftyOne plugins differ from viewer extensions Most robotics viewers have extension points, and they're usually scoped to the recording you have open: a custom panel, a message converter, or a new file loader that runs in the browser or desktop app. Labeling platforms add hooks that act on one task in a workflow stage. FiftyOne plugins run on the server, in Python, against any view of a dataset. A single plugin can add a form, run a job across every episode in the background, render a custom panel with the results, and expose all of it to agents through the MCP server, without leaving the language your robotics stack is written in. For live debugging of a single recording, a dedicated log viewer is the right tool, and FiftyOne doesn't try to replace it. FiftyOne reads .rrd files and MCAP files, so a common setup is to debug individual logs in a viewer and curate the dataset in FiftyOne. Two FiftyOne plugins for MCAP and LeRobot curation Both plugins below were built using only the pieces covered above. Both follow the same rule, stated in their READMEs: the scores order your review queue, and a person makes the call. A perfectly smooth demonstration can still show the wrong task. LeRobot Data Curation lerobot_data_curation ranks the episodes of a LeRobot v3 dataset worst-first, so you know which to look at before training a policy or fine-tuning a VLA. It scores each episode on motion smoothness, time efficiency, action-state tracking, gripper behavior, consistency, integrity, and language, and optionally on camera quality: blur, exposure, clipping, frozen feeds, and video-action lag. It has no model-based metrics, so there's no model to download. How it maps to the framework: - A typed form with tabs. The compute operator's form has Data, Metrics, Camera, and Normalization tabs. You pick the observation.state and action arrays, and assign arm and gripper joints by name. Nothing is guessed from the data, and your picks are remembered for the next run. - Delegated by default. The operator sets allow_delegated_execution=True and default_choice_to_delegated=True, so scoring runs in the background. Camera decoding is the slow part. - A JS panel backed by Python operators. The LeRobot Curation panel is a React component with Overview, Motion & Action, Integrity & Coverage, Vision and Language tabs. Click a histogram bar to filter the grid to those episodes. Click a row to open an inspector with joint traces, the speed profile, the gripper timeline, and frames picked around the flagged spans. - It follows the view. The panel refreshes when the view changes, so filtering the grid re-ranks it. - Tags on the timeline. Flagged spans such as idle stretches, the longest pause, and acceleration spikes are written as temporal tags on each episode so that you can scrub straight to them in the multimodal viewer. Footer buttons apply review, exclude-candidate, and relabel sample tags. import fiftyone as fo dataset = fo.Dataset.from_dir( dataset_dir="/data/lerobot/my_dataset", dataset_type=fo.types.LeRobotDataset, ) session = fo.launch_app(dataset) # then run "LeRobot curation: compute quality" from the operator browser Keep it honest. Scores are relative to the batch you scored. Per-task normalization needs about 20 episodes per task, and the default thresholds were calibrated on a 102-episode development set. Check them on your own data. Cloud-hosted LeRobot sources aren't supported yet. Demo Quality Scorer demo_quality_scorer does the same job for multimodal MCAP datasets: teleoperations, autonomous vehicles, drones, rovers, and egocentric capture. !Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer. Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer. It computes three metric families and ranks episodes worst-first: 1. Motion smoothness: SPARC, log dimensionless jerk (LDLJ), jerk RMS, and a low-to-high frequency power ratio, computed over windows of a speed profile you choose from position or velocity signals such as /odom -> twist_linear. 1. Sensor health: dropout, rate stability, clock drift, clipping, and cross-channel desync, all from message timestamps and raw values, so it works on any channel the plugin can decode. 1. Outliers: models that flag episodes whose motion and health features don't look like the rest of the batch. How it maps to the framework: - A dynamic form. The form inspects the first sample's channels. If no telemetry channel carries a numeric signal, the motion family switches itself off and says why. If the server lacks a decoder, the form names the package to install. - Immediate or delegated, your choice. Run immediately with an in-app progress bar, or schedule it in the background (default). - A JS panel. The Episode Quality panel is a React component that has Motion, Health, and Outliers tabs. Click a histogram bar to filter the grid. Click a table row to open that episode in the multimodal viewer, with a toast naming the timecode of its worst flagged interval. - Results as fields. Scores land in quality.* fields such as quality.sparc and quality.ldlj, so you can sort and filter on them outside the panel too. pip install "fiftyone[multimodal]>=1.22.0" mcap mcap-protobuf-support \ mcap-ros1-support mcap-ros2-support numpy scipy scikit-learn fiftyone plugins download https://github.com/harpreetsahota204/demo_quality_scorer Keep it honest. Structural errors, such as the wrong action at a key moment, are invisible to every metric here. Re-running over a different set of episodes re-fits normalization, so expect scores to move. Build your own in six steps Your robot has checks neither plugin covers. Here's the loop for writing them. !Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write. Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write. Step 1: Scaffold the plugin Run fiftyone plugins create or make a folder in your plugins directory with a fiftyone.yml: name: "@your-org/gripper_checks" description: Flags episodes with repeated grasp attempts operators: - score_grasps Step 2: Write the operator One class per operator or panel, like the examples above. Keep the scoring logic in a plain Python function so a notebook, a test, and the operator can all call it. Step 3: Register it Call p.register(ScoreGrasps) inside register(p), and make sure the name matches fiftyone.yml. Step 4: Run the App in debug mode export VFF_MULTIMODAL=1 fiftyone app debug my_lerobot_dataset fiftyone app debug prints server logs and tracebacks to your terminal. VFF_MULTIMODAL=1 turns on the multimodal viewer in the App; set it before fiftyone is imported in every process that touches a multimodal dataset. Step 5: Iterate Edit, refresh the App, repeat. Python changes are picked up on the next request without a server restart. If you have a React panel, keep npm run dev rebuilding it. Step 6: Ship it Push to GitHub. Anyone installs it with: fiftyone plugins download https://github.com/your-org/gripper_checks fiftyone plugins requirements @your-org/gripper_checks --install Good to know. How do I test an operator without the App? Operators are callable from Python with foo.execute_operator, so you can test them headless in pytest. Try it yourself - Plugin developer guide - FiftyOne multimodal user guide for MCAP and LeRobot - voxel51/fiftyone-plugins, the official plugins and panel examples - lerobot_data_curation - demo_quality_scorer - FiftyOne MCP server - VOODO, the React design system for plugin panels Frequently Asked Questions How do I curate a LeRobot dataset before training a policy? Import it with fo.Dataset.from_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset), where each sample is one episode. Then install lerobot_data_curation and run its compute operator to rank episodes worst-first on motion, gripper, camera, and language metrics. Review the top of the list and tag episodes as exclude-candidate or relabel. Can FiftyOne open MCAP files? Yes. FiftyOne treats .mcap, .bag, and .rrd files as media_type="multimodal" samples, one episode per file, and plays their streams in a synchronized multimodal viewer. Set VFF_MULTIMODAL=1 before importing fiftyone to turn the viewer on. How do I find bad demonstrations in robot teleoperation data? Score every episode on motion smoothness (SPARC, LDLJ, jerk) and sensor health (dropout, rate stability, clock drift), then sort worst-first. demo_quality_scorer does this for MCAP datasets. Treat the scores as a review queue, because a smooth demonstration can still show the wrong task. Do FiftyOne plugins need JavaScript? No. A Python-only plugin is fiftyone.yml plus __init__.py, and the App builds forms and panels from Python declarations. Add React only when you need custom interactivity, such as the dashboards in lerobot_data_curation and demo_quality_scorer. Where does FiftyOne plugin code run? It runs on the FiftyOne server, in Python, or on a background worker for delegated operations. It never runs in the browser, so it can use any Python library, a GPU, or secrets without exposing them.

50 Embodiments, 497 Episodes, One Indexed Dataset: Curating the LeRobot Community Dataset with FiftyOne A single prompt to an AI agent running FiftyOne Skills pulled 10 episodes from every robot embodiment in the LeRobot community dataset, computed video embeddings with Qwen3-VL, and built a queryable similarity index. Here's the code and the thinking behind it. The LeRobot community dataset is one of the largest community-built pools of cross-embodiment robot data: 235 contributors, 50 robot types, ~900 GB. Hugging Face's dataset card lists 46 robot types across 791 cleaned datasets. The 50 here come from scanning robot_type in all 1,755 contributor folders, so the two counts cover different sets of datasets. It's organized as 1,755 contributor datasets, one per folder, so picking a balanced slice starts with reading each one's metadata. If you're training a policy that has to generalize across robots, you need episodes from every embodiment and a way to see which clips are actually worth keeping. Now that FiftyOne natively supports LeRobot, I figured I’d use the FiftyOne skill to import LeRobot datasets and check it out. All it took was a one-sentence prompt: "i want to have a little bit from every embodiment, i have enough disk space for the whole dataset... but let's take 10 episodes from each embodiment to start" That sentence kicked off a session that read every contributor dataset's metadata, found 50 unique robot types, downloaded the right shard files for each, imported 497 episodes into FiftyOne, embedded every clip with Qwen3-VL, computed a similarity index, and saved 75 queryable views. All without writing a data loader, a Parquet parser, or a video decode loop. One sentence was enough because the agent had FiftyOne Skills installed. Skills are packaged workflows that teach AI assistants how to do complex FiftyOne tasks: importing LeRobot episodes, running zoo models, computing embeddings, finding duplicates. The agent didn't have to guess at the LeRobot v3 layout or the FiftyOne API. It followed the skill. Install them in Cursor, Claude Code, Codex, or Gemini CLI with one command: curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills This is what FiftyOne is for. Not visualization. Dataset curation at the scale that robot learning actually requires. New here? FiftyOne is an open-source toolkit for loading, querying, visualizing, and curating ML datasets — images, videos, and multimodal data. LeRobot is Hugging Face's framework for robot learning: it defines a standard v3 dataset format (Parquet + MP4 shards with episode metadata) that the community uses to share teleoperation recordings. Key takeaways - lerobot/community_dataset_v3 is a collection of 1,755 contributor datasets across 50 robot embodiments, each a complete LeRobot v3 dataset with its own meta/info.json. - Using the FiftyOne Skills dataset-import workflow, the agent scanned every */*/meta/info.json, grouped by robot_type, picked the smallest source dataset per embodiment with ≥10 episodes, and downloaded only the needed shard-0 files. - Each FiftyOne sample is one episode — not a frame, not a video file — with fields robot_type, task, duration, fps, dataset_name, and a media_reference pointer into the local LeRobot directory tree. - Qwen3-VL-Embedding-2B turns each episode (up to 32 frames from its time window) into one 2048-dimensional video embedding. The similarity index, Uniform Manifold Approximation and Projection (UMAP), uniqueness, and representativeness all run from that one field in under a minute. - The same model embeds text into the same space, so the index answers text queries: type "folding a cloth" and get the same bimanual cloth-folding episodes back, with no labels involved. - uniqueness scores range from 0.013 to 1.0 across the 497 episodes. 292 clips score below 0.1 because each embodiment's 10 clips come from one recording session. The two highest scores, the only ones above 0.7, both belong to single-frame episodes (0.02–0.03 s long): uniqueness flags broken recordings as readily as rare behavior. - 75 saved views — 50 by robot type plus analytical cuts by frames per second (FPS), duration, robot category, task keyword, and curation intent — are stored in the dataset and load from the App's Views dropdown with no code. What you need To load and query the pre-built dataset (no GPU required): pip install "fiftyone>=1.22.0" "huggingface_hub[cli]" transformers qwen-vl-utils torch torchvision To re-run UMAP visualization yourself (Step 3): pip install umap-learn FiftyOne 1.22.0 is the first release with fo.types.LeRobotDataset. The dataset loads from the Hub with embeddings, similarity index, UMAP layout, and curation scores already computed. Steps 1–2, Step 4, and episode-to-episode search in Step 5 run entirely on the pre-built data, with no GPU or model weights. Text search in Step 5 downloads Qwen3-VL-Embedding-2B (about 4 GB) to embed your query; it runs on CPU and is faster on a GPU. Step 3 is for readers building their own dataset who want to replicate the embedding pipeline. How the LeRobot community dataset is organized !FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values lerobot/community_dataset_v3 is a collection of contributor datasets. Each one is a complete LeRobot v3 dataset in its own <contributor>/<dataset>/ folder: community_dataset_v3/ kuehnrobin/ g1_stack_cube_left/ meta/info.json data/ videos/ villekuosmanen/ pick_coffee_capsule_under_dome/ meta/info.json ... 1,755 directories. 17,617 files. 897.5 GB total. Each sub-dataset carries its own meta/info.json with its robot type, episode count, and FPS. fo.types.LeRobotDataset reads one LeRobot v3 dataset at a time, so the agent started by downloading all */*/meta/info.json files — a 56-second fetch — and then scanning them to build the plan: # Counts across all 1,755 sub-datasets Robot types (50 unique): so100 899 datasets 34801 total_episodes so101 312 datasets 7490 total_episodes so101_follower 121 datasets 4414 total_episodes so100_follower 120 datasets 3065 total_episodes arx5 43 datasets 2108 total_episodes Unitree_G1_inspire 1 dataset 7 total_episodes # only 7 available so100MovellaDot 75 datasets 157 total_episodes # max 5 per source ... All 1,755 sub-datasets were already at codebase_version: v3.0. No conversion needed. !Diagram of the contributor and dataset folder layout, the four-step plan to scan, group by robot type, pick one source per embodiment, and download, and the shard-0 Parquet and MP4 files fetched from one source Left: the contributor/dataset folder layout, with the per-dataset meta/info.json files the agent scans first. Middle: the four-step plan — scan, group by robot_type, pick one source dataset per embodiment, download. Right: inside one source dataset, only file-000.parquet and file-000.mp4 (the first ~10 episodes) are fetched; the other shards are skipped. Good to know. What's a shard-0 set? LeRobot v3 packs many episodes into shared MP4 and Parquet shards. file-000.parquet and file-000.mp4 hold the first batch of episodes for each dataset. Downloading only those files makes a 10-episode subset cheap, even from a 50-GB source dataset. Step 1: load the dataset The agent session did the heavy lifting — reading every contributor dataset's metadata, picking the right shard files, importing 497 episodes, computing embeddings, and pushing the result to the Hub. The import step is covered in How to Import LeRobot Datasets into FiftyOne Using Agent Skills. The dataset is at Voxel51/community_v3_10per_embodiment. Load it: import fiftyone as fo import fiftyone.utils.huggingface as fouh ds = fouh.load_from_hub("Voxel51/community_v3_10per_embodiment") print(len(ds)) # 497 print(ds.distinct("robot_type")) # 50 embodiments print(ds.get_field_schema()) # episode_index, robot_type, task, duration, fps, # dataset_name, media_reference, # qwen3vl_embedding, qwen3vl_embedding_camera, # uniqueness, representativeness, ... From here on, the source videos live in the Hub repo alongside the metadata. FiftyOne reads through the media_reference pointer — nothing to download separately. And because the unit is an episode rather than a file, every query you run is a dataset-level question, not a file-browser search. What "one sample = one episode" actually means !A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference Most tools treat robot data as video files or frame sequences. FiftyOne takes a different unit: the episode. Every sample in this dataset carries: sample.episode_index # int — 0..9 within the source dataset sample.robot_type # str — e.g. "arx5", "so100", "Unitree_G1_Dex3" sample.dataset_name # str — e.g. "villekuosmanen/pick_coffee_capsule_under_dome" sample.task # str — e.g. "Pick up the coffee capsule" sample.duration # float — seconds (0.02 to 148.6 in this dataset) sample.fps # float — 10, 15, 20, 25, 30, or 50 sample.media_reference # LeRobotEpisodeReference — a FiftyOne type with two fields: # .data = [chunk_index, file_index, first_row, last_row] (Parquet) # .videos = {camera: [chunk_index, file_index, from_ts, to_ts]} (MP4) Think of a FiftyOne sample as a row in a table: the left side has filterable scalar fields (robot_type, task, duration, fps) and the right side has the media_reference pointer. The pointer's two values — data[2:4] and videos[top][2:4] — map to the actual Parquet rows and MP4 time window in the LeRobot folder on disk. !Diagram of a FiftyOne sample with filterable scalar fields above a media_reference pointer whose data and videos values map to Parquet rows and an MP4 time window on disk A FiftyOne sample (left) with filterable scalars above the line and the media_reference pointer below it. Arrows connect the pointer's two values — data[2:4] and videos[top][2:4] — to the actual Parquet rows and MP4 time window in the LeRobot folder on disk. Three roads lead from a FiftyOne view back to frame data: !Diagram of three ways to get frames from a FiftyOne view: read in place and Torch dataset, which copy no data, and export to a standalone LeRobot v3 dataset, which copies data 1. Read in place via sample.media_reference.data — no copy, your code, one episode at a time. 1. Torch dataset via view.values("episode_index") passed to LeRobotDataset(..., episodes=...) — no copy, for training loops. episode_index restarts at 0 in every source dataset, so filter the view to one dataset_name first; for a mix of sources, export instead 1. Export via view.export(dataset_type=fo.types.LeRobotDataset) — copies data, produces a standalone v3 repo. Each export must come from a single source dataset, so filter by dataset_name first. The rule: in-process → don't copy. Out-of-process → copy. | Path | Call | Copies data? | Use it when | | --- | --- | --- | --- | | Read in place | sample.media_reference.data | No | Your own code needs one episode's rows, such as for a plot, a metric, or a policy | | Torch dataset | LeRobotDataset(..., episodes=view.values("episode_index")) | No | A training loop reads batches from a curated subset filtered to one dataset_name | | Export | view.export(dataset_type=fo.types.LeRobotDataset) | Yes | Something outside the Python process needs a standalone LeRobot v3 repo, one source per export | Three ways to get frames from a FiftyOne view of LeRobot episodes, and when each one copies data From inspection to curation !Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid Timeline viewers are built for inspecting one recording at a time: open a log, scrub the timeline, and watch camera, inertial measurement unit (IMU), and joint streams in sync. That's the right tool when you're debugging one specific run. FiftyOne starts where that ends. Once a run is recorded and you have hundreds of episodes, the question isn't "what happened in this episode?" It's "across all 497 episodes from 50 different robots, which clips are visually unique, which are near-duplicates, which ones actually belong in my training set?" A timeline scrubber can't answer that. A queryable, indexed dataset can. The rest of this post shows exactly what that looks like — every step from here on is something you can't do in a timeline scrubber. Step 2: what's already on the dataset The Hub dataset ships with a video embedding for every episode, computed with Qwen/Qwen3-VL-Embedding-2B from the qwen3vl_embeddings remote zoo source. Each episode's time window is decoded with PyAV; up to 32 evenly spaced frames go to the model as one video clip, producing a 2048-dimensional qwen3vl_embedding field. qwen3vl_embedding_camera records which camera stream the frames came from, since camera names and layouts differ across the 50 embodiments. What Qwen3-VL-Embedding actually produces, and why it matters Qwen3-VL is an open vision-language model family from Qwen/Alibaba. It comes in two variants: Qwen3-VL-Instruct (generates text — captions, bounding boxes, answers) and Qwen3-VL-Embedding (trained for retrieval, so it produces dense vectors directly). The embedding model reads the clip, and the hidden state of the last token becomes the vector, L2-normalized: # what produced qwen3vl_embedding (inside the qwen3vl_embeddings zoo model) outputs = model(**inputs) # frames + instruction embedding = outputs.last_hidden_state[row, last_token_position] # one 2048-d vector per clip embedding = F.normalize(embedding, p=2, dim=-1) !Two-lane diagram in which frames decoded with PyAV pass through Qwen3-VL-Embedding into one 2048-dimensional vector, and a text query passes through the same model into the same vector space Top lane: an episode's time window is decoded with PyAV, up to 32 evenly spaced frames are tokenized by the vision encoder and run through the language model, and the last token's hidden state becomes one 2048-d vector stored as qwen3vl_embedding. Bottom lane: at query time, a text query goes through the same model and lands in the same space, which is what makes text-to-video search work. This model is useful for robotics data because video and text map into the same vector space. The similarity index qwen3vl_embedding_sim keeps the model name, so it accepts text queries as well as episodes: ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim") — no labels, no metadata matching, just the model's understanding of what was in the clip. For Vision-Language-Action (VLA) model development, this is specifically useful at the data curation stage. A VLA policy needs to generalize across scenes, embodiments, and task variations. But you can only curate what you can measure. Video embeddings give you a numeric handle on task semantics and visual appearance simultaneously, which makes it possible to compute: - Similarity — which episodes, across all 50 embodiments, are doing visually similar things? - Uniqueness — which demonstrations cover behavior your dataset hasn't seen much of? - Representativeness — which episodes best stand in for entire clusters of similar behavior? You can't extract these signals from episode metadata alone. They require a model that actually watched the clips. You also get uniqueness and representativeness scores out of the box. More on those below. If you want to try a different model, such as Qwen/Qwen3-VL-Embedding-8B, write its vectors to a new field; Step 5 shows the full embedding loop for LeRobot episodes. Step 3: Compute similarity, uniqueness, and representativeness !FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram Skip the code if you loaded from the Hub. The pre-built dataset already includes qwen3vl_embedding_sim, qwen3vl_embedding_umap, uniqueness, and representativeness. The code block below is for readers building their own LeRobot dataset — it requires umap-learn and an existing embedding field. The explanations after it apply either way. If you're building your own dataset, compute the qwen3vl_embedding field first (the loop is in Step 5), then run the four calls: import fiftyone.brain as fob EMBED = "qwen3vl_embedding" # Similarity index — powers Sort by Similarity in the App. Passing the model name # lets the index embed text queries (register the qwen3vl_embeddings source first) fob.compute_similarity( ds, embeddings=EMBED, model="Qwen/Qwen3-VL-Embedding-2B", brain_key="qwen3vl_embedding_sim", ) # UMAP — 2D layout for the Embeddings panel fob.compute_visualization( ds, embeddings=EMBED, method="umap", brain_key="qwen3vl_embedding_umap", num_dims=2, seed=51, ) # Uniqueness — how different each episode is from its nearest neighbors fob.compute_uniqueness( ds, embeddings=EMBED, uniqueness_field="uniqueness", similarity_index="qwen3vl_embedding_sim", ) # Representativeness — how close each episode is to a cluster center fob.compute_representativeness( ds, embeddings=EMBED, representativeness_field="representativeness", similarity_index="qwen3vl_embedding_sim", ) Why UMAP for video embeddings? Each episode is a 2048-dimensional vector. You can't look at that. Dimensionality reduction maps it to 2D so you can see structure in the Embeddings panel. The three common choices are principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and UMAP: - PCA is linear. It finds the axes of maximum variance, which is fast but can't capture the curved manifold structure that emerges when you embed semantically diverse video. A pick-and-place cluster and a folding cluster may sit on a low-variance direction PCA ignores entirely. - t-SNE is non-linear and good at revealing local clusters, but it destroys global structure — the distance between clusters in the 2D plot is meaningless, and it's slow on datasets larger than a few thousand samples. - UMAP is non-linear, keeps local cluster structure, and usually keeps more of the arrangement between clusters than t-SNE does. Don't over-read it, though: distances between far-apart clusters are still only roughly meaningful. It runs on 500 episodes in seconds. FiftyOne's UMAP backend uses Euclidean distance with min_dist=0.1 by default, which gives tight clusters. !Three scatter plots of the same 497 Qwen3-VL-Embedding vectors reduced with PCA, t-SNE, and UMAP, colored by robot category The same 497 Qwen3-VL-Embedding vectors reduced three ways, colored by robot category. PCA's two axes hold 20% of the variance, and the categories overlap; t-SNE separates local knots but inter-knot distances are arbitrary; UMAP is the layout the Embeddings panel shows. In every map, the 10 clips from one source dataset collapse into a single knot—same scene, same camera, same task. For a first look at a cross-embodiment dataset, UMAP is the practical choice: you get clusters you can click into, and their rough arrangement is meaningful. The map also shows what the embeddings respond to most: the recording session. For 495 of the 497 episodes, the nearest neighbor comes from the same source dataset. Step past a clip's own siblings, though, and the next matches in the full 2048-d space often share the task on a different robot. The nearest non-sibling to the so100 box-in-bin episodes is a so100MovellaDot ball-in-box episode (cosine distance 0.157), and the nearest non-sibling to the arx5 dome-lifting episodes is a trossen_ai_stationary cube-in-box episode (0.170). What uniqueness and representativeness actually compute Both metrics are grounded in the embedding space. The FiftyOne Brain docs cover uniqueness and representativeness; here's what each one computes: Uniqueness — K-nearest-neighbor distance, K=3. For each episode, FiftyOne finds its 3 nearest neighbors in embedding space and computes a weighted mean of those distances: uniqueness = weighted_mean(distances_to_3_nearest_neighbors) weights = [0.6, 0.3, 0.1] # closer neighbors count more normalized to [0, 1] by dividing by the dataset maximum Interpretation: uniqueness=1.0 means this episode is farther from its nearest neighbors than any other episode in the set — it occupies a region of embedding space that no other clip is near. uniqueness=0.013 means the episode is almost on top of another one — effectively a near-duplicate. !Toy embedding space where an isolated episode scores 1.0 and a clustered one scores about 0.01 on uniqueness, beside a log-scale histogram of uniqueness across 497 LeRobot episodes Left: a toy embedding space. Episode A is far from its three nearest neighbors (line width = weight), so it scores 1.0; episode B sits in a knot of near-identical clips and scores ≈ 0.01. Right: the real distribution over all 497 episodes on a log scale — 292 score below 0.1, two score above 0.7, and one arx5 clip (a single-frame episode) sets the maximum at 1.0. Representativeness — K-Means distance to cluster center, K=20 clusters. For each episode, FiftyOne finds the nearest cluster center and scores it as: representativeness = 1 / (1 + distance_to_nearest_cluster_center) normalized locally within each cluster Interpretation: representativeness=1.0 means this episode is at the heart of its cluster — it's the most "typical" example of that group of behaviors. Low representativeness means the episode sits at the edge of a cluster, more idiosyncratic than central. !K-means cluster centers with equal-distance rings, where a central episode scores near 1.0 and a peripheral one near 0.3, beside the representativeness distribution across 497 episodes Left: K-Means centers (×) with rings of equal distance. Episode P sits on its center and scores ≈ 1.0; episode Q is assigned to the blue cluster but sits far out and scores ≈ 0.3. Right: the real distribution — 198 of 497 episodes score above 0.8, and the floor of 0.33 means no episode is a pure outlier. Why these two metrics are the right pair for VLA data curation A Vision-Language-Action model needs two things from its training data: 1. Coverage — enough behavioral diversity for the policy to generalize to new scenes and task variations. Low-uniqueness episodes (near-duplicates) don't add coverage; they just add training compute and risk overfitting to that one scene configuration. 1. Signal density — enough repetition of each important behavior for the policy to learn it. Keeping only high-uniqueness episodes would discard the repetitions that make a skill learnable. Uniqueness and representativeness together let you navigate that tradeoff explicitly: | Goal | Query | | --- | --- | | Seed set: diverse AND learnable | High uniqueness + high representativeness | | Redundancy audit: what can be cut? | Low uniqueness floor (near-duplicates) | | Canonical demos per task | Sort by representativeness within each task view | | Rare demonstrations (or broken ones) | Uniqueness > 0.7, then check duration | Four VLA data curation goals and the uniqueness and representativeness queries that answer each one. !Scatter plot of uniqueness against representativeness for every LeRobot episode, colored by robot category, with shaded query regions and 50 ringed points marking curate_diverse_seeds Every episode is placed by its two scores (uniqueness on a symlog axis, colored by robot category). The shaded regions are the queries from the table: the redundancy audit on the left, canonical demos along the top, high-uniqueness outliers on the right, and the 50 orange-ringed points are curate_diverse_seeds — the top-50 by uniqueness with representativeness above 0.5. In this dataset, uniqueness ranges from 0.013 to 1.0. The floor shows near-identical clips in the set — 292 of 497 score below 0.1, and the bottom six are four moss cube-stacking episodes and two sam_bimanual cloth-folding episodes, each from a single recording session. At the top, only two episodes score above 0.7: an arx5 clip at 1.0 and an xarm6 clip at 0.86. Both are single-frame episodes, 0.02–0.03 seconds long. representativeness ranges from 0.33 → 1.0 — no episode is a pure outlier; even the most peripheral clips are loosely connected to a cluster. Keep it honest. Both ends of the uniqueness ranking come from how we built this subset, not from rare behavior or wasted data. Each embodiment contributes 10 consecutive episodes from one recording session, so every clip has nine siblings sitting right next to it in embedding space. On this dataset, every one of the 50 episodes in curate_redundant_candidates has a sibling as its nearest neighbor, sampled on purpose. The top of the ranking has the opposite story: the five single-frame episodes look unlike everything else because there's almost nothing in them. They take the top two spots, and all five land in curate_diverse_seeds. Uniqueness did its job by surfacing recordings worth a second look. Those clips belong in the cut pile, though, not the seed set, so pair uniqueness with a duration filter. These two numbers let you make dataset decisions — what to keep, what to cut, what's overrepresented — without watching a single video. A log viewer shows you one episode at a time. FiftyOne gives you a signal across all 497 at once. Step 4: Save views that answer real questions !Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App Views in FiftyOne are lazy queries — no data is copied, they resolve on load. Once saved, they appear in the App's Views dropdown. The pre-built dataset already has 75 views saved. Check what's there: import fiftyone as fo print(ds.list_saved_views()) # ['so100', 'so101', ..., 'curate_diverse_seeds', 'curate_redundant_candidates', # 'category_humanoid', 'task_pick_place', 'fps_30', ...] To build them yourself on your own dataset: import fiftyone as fo # Curation: 50 most visually distinct AND cluster-representative episodes ds.save_view( "curate_diverse_seeds", ds.sort_by("uniqueness", reverse=True) .match(fo.ViewField("representativeness") > 0.5) .limit(50), ) # Redundancy candidates: bottom 50 by uniqueness ds.save_view( "curate_redundant_candidates", ds.sort_by("uniqueness").limit(50), ) # Robot category views HUMANOID = ["Unitree_G1_Dex3", "Unitree_G1_Gripper", "Unitree_G1_Inspire", "Unitree_G1_inspire", "g1"] ds.save_view("category_humanoid", ds.match(fo.ViewField("robot_type").is_in(HUMANOID))) # Task keyword views ds.save_view("task_pick_place", ds.match(fo.ViewField("task").re_match("pick|place|grasp|grab", options="i"))) # FPS views — reveal recording-condition variation across embodiments for fps_val in [10.0, 15.0, 20.0, 25.0, 30.0, 50.0]: view = ds.match(fo.ViewField("fps") == fps_val) if len(view) > 0: ds.save_view(f"fps_{int(fps_val)}", view) 75 views total. fps_30 covers 387 of the 497 episodes. fps_10 covers 30 — all from three stationary arms: widowx, ur5e_gello, and mycobot320. task_pick_place returns 367 episodes; task_fold_pour returns 70. Each is a reusable, shareable slice of the dataset that resolves in milliseconds. No script, no re-download, no custom data loader — just a name in a dropdown. Step 5: similarity search by episode and by text !Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel Search by episode The qwen3vl_embedding_sim index powers episode-to-episode search directly from the App or from Python. Pass any sample's ID, and FiftyOne returns the nearest neighbors by video embedding distance. Because every clip's closest matches are its own recording session, filter out the query's dataset_name to see what other robots are doing something similar: from fiftyone import ViewField as F # one sam_bimanual cloth-folding episode as the query query = ds.match( (F("dataset_name") == "girardijp/sam_fold_cloth_single") & (F("episode_index") == 0) ).first() # unfiltered: the query and its 9 siblings from the same session same_session = ds.sort_by_similarity(query.id, k=10, brain_key="qwen3vl_embedding_sim") # filtered: the 10 nearest episodes from other source datasets others = ( ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim") .match(F("dataset_name") != query.dataset_name) .limit(10) ) for s in others: print(f"[{s.robot_type}] {s.task}") # [so100_bimanual] playing with folding (all 10 hits) !A sam_bimanual cloth-folding query episode marked on the UMAP, beside the 10 nearest episodes from other source datasets ranked by cosine distance, all so100_bimanual folding clips Left: the query (star) on the UMAP, with its session's knot and the cross-source neighbors. Right: the 10 nearest episodes from other source datasets, ranked by cosine distance. The query's 9 siblings all sit within 0.015; past them, every hit is a so100_bimanual "playing with folding" episode, 0.19–0.31 away. !Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel A sam_bimanual rig folding a cloth and a pair of SO-100 arms folding cloth look nothing alike as robots, and the model still puts them next to each other. That's the cross-embodiment signal. Some sessions pull in a neighboring robot type instead: the 10 Unitree_G1_Dex3 cube-stacking clips sit within 0.030 of each other, and the first other episode, at rank 10, is a Unitree_Z1_Dual red-cup pick (0.142). Search by text !Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes The same index takes text. The dataset stores the episode vectors, but FiftyOne still needs the model to embed your query, so register the qwen3vl_embeddings remote zoo source first: import fiftyone.zoo as foz # declare which models the remote source provides foz.register_zoo_model_source( "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True, ) # optional: the first text query downloads the weights automatically foz.download_zoo_model("Qwen/Qwen3-VL-Embedding-2B") for query in ["folding a cloth", "pouring water into a cup", "humanoid robot picking up a can"]: view = ds.sort_by_similarity(query, k=3, brain_key="qwen3vl_embedding_sim") print(query, "->", [(s.robot_type, s.task) for s in view]) # summarised output: # folding a cloth -> sam_bimanual, "Fold a cloth" (x3) # pouring water into a cup -> bi_so100_follower, "bi soarm100 pour water" (x3) # humanoid robot picking up a can -> Unitree_G1_Inspire, "Pick up the cans into the box on the table." (x3) The same text box works in the App's similarity panel once qwen3vl_embedding_sim is selected. Not every query lands: "robot stacking a cube" returns a Unitree_Z1_Single red-cup pick rather than the Unitree_G1_Dex3 cube-stacking clips. Treat text search as a fast way to find candidates, then check them. Build the embedding and index on your own dataset LeRobot samples have no filepath, so compute_embeddings can't read the clips for you: decode each episode's time window with PyAV and pass the frames to the model as one video. This is the code behind qwen3vl_embedding (set ROOT to the folder that holds your <contributor>/<dataset>/ directories): from pathlib import Path import av import fiftyone.brain as fob import fiftyone.zoo as foz ROOT = Path("/path/to/community_dataset_v3") def episode_frames(video_path, from_ts, to_ts, max_frames=32): frames = [] with av.open(str(video_path)) as container: stream = container.streams.video[0] fps = float(stream.average_rate or 30) step = max(1, int((to_ts - from_ts) * fps) // max_frames) container.seek(int(from_ts * 1_000_000), any_frame=False, backward=True) i = 0 for frame in container.decode(stream): ts = float(frame.pts * stream.time_base) if ts < from_ts: continue if ts > to_ts or len(frames) >= max_frames: break if i % step == 0: frames.append(frame.to_image()) i += 1 return frames foz.register_zoo_model_source( "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True ) model = foz.load_zoo_model("Qwen/Qwen3-VL-Embedding-2B") # or Qwen/Qwen3-VL-Embedding-8B if model._embedder is None: model._load_model() for sample in ds.iter_samples(autosave=True, progress=True): cam, (chunk, file, from_ts, to_ts) = next(iter(sample.media_reference.videos.items())) path = ROOT / sample.dataset_name / "videos" / cam / f"chunk-{chunk:03d}" / f"file-{file:03d}.mp4" frames = episode_frames(path, from_ts, to_ts) # model.embed() expects a file path; the underlying embedder accepts a frame list as a video sample["qwen3vl_embedding"] = model._embedder.process([{"video": frames}])[0].cpu().float().numpy() sample["qwen3vl_embedding_camera"] = cam fob.compute_similarity( ds, embeddings="qwen3vl_embedding", model="Qwen/Qwen3-VL-Embedding-2B", brain_key="qwen3vl_embedding_sim", ) Passing the model name to compute_similarity lets the index embed text at query time; an index built without it accepts only episodes as queries. On a GPU, the 497 episodes take about 20 minutes. Then run the UMAP, uniqueness, and representativeness calls from Step 3. Putting it together: export a curated LeRobot v3 training split Once loaded, querying across 50 embodiments looks like this: # 50 most unique AND representative episodes, minus the single-frame recordings seed_view = ds.load_saved_view("curate_diverse_seeds").match(fo.ViewField("duration") > 1) # Episodes from other robots that look like a given clip query = ds.match(fo.ViewField("robot_type") == "sam_bimanual").first() similar = ( ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim") .match(fo.ViewField("dataset_name") != query.dataset_name) .limit(10) ) # Episodes matching a description (register the qwen3vl_embeddings source first) cloth = ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim") # Export the curated subset as standalone LeRobot v3 repos, one per source dataset for name in seed_view.distinct("dataset_name"): seed_view.match(fo.ViewField("dataset_name") == name).export( export_dir=f"./my_training_subset/{name.replace('/', '__')}", dataset_type=fo.types.LeRobotDataset, ) The export loops over dataset_name because a LeRobot v3 repo has one feature schema, and the 50 embodiments here have different cameras and action dimensions. The 45 seed episodes come from 16 source datasets, so you get 16 repos. That last step is the point. You went from a 900 GB community dataset to a curated, re-indexed LeRobot v3 training split — filtered by visual uniqueness, robot category, task type, or any combination — without writing a parser, a data loader, or a custom video decode loop. A viewer shows you what the robot did. A curated dataset decides what goes into the model. Frequently Asked Questions What does fo.types.LeRobotDataset actually store in FiftyOne's database? Episode-level metadata only: episode_index, task, duration, fps, robot_type, and a media_reference that points into the local LeRobot directory. No pixel data, no Parquet rows. The source files stay where they are; FiftyOne reads through the pointer. Can I use apply_model on a LeRobot dataset in FiftyOne? Not for a standard zoo model — those expect sample.filepath pointing to a video or image file, and LeRobot samples use media_reference instead. The embedding in this post was computed by working around that: PyAV extracts frames from the episode's time window, and the frames go to the model as a list of PIL images. Step 5 shows the full loop. Why does Qwen3-VL-Embedding use last-token pooling? That's how Qwen3-VL-Embedding was trained. It's a decoder-only model with no CLS token, and its retrieval training teaches the final token's hidden state to summarize the whole input, whether it's a video clip or a sentence. Using the same pooling for both is what puts episodes and text queries in one space. Mean pooling over an instruction-tuned model's hidden states gives usable vectors for comparing clips, but not for matching them against text. How do saved views work after load_from_hub? They're stored in the FiftyOne dataset document alongside the samples. ds.list_saved_views() returns all 75 names; ds.load_saved_view("curate_diverse_seeds") returns the 50 episodes matching that query, resolved against whatever embeddings and fields exist locally. How do I export a curated FiftyOne view to LeRobot v3 for policy training? Export a curated view as standalone LeRobot v3 repos, one per source dataset: for name in view.distinct("dataset_name"): view.match(fo.ViewField("dataset_name") == name).export( export_dir=f"./my_curated_subset/{name.replace('/', '__')}", dataset_type=fo.types.LeRobotDataset, ) The exporter refuses to mix episodes from different sources in one repo, since each embodiment has its own cameras and action layout. For each source, it re-indexes episodes from 0, recomputes stats, and writes a self-contained v3 structure that any LeRobot-compatible trainer can read directly.

How to Import LeRobot Datasets into FiftyOne Using Agent Skills !FiftyOne App showing a LeRobot dataset of robot arm episodes, with the sidebar filtering 800 samples down to 100 by task. FiftyOne App showing a LeRobot dataset of robot arm episodes, with the sidebar filtering 800 samples down to 100 by task. A standardized dataset format solved robot learning’s data fragmentation problem. Here’s how to get one of those datasets into FiftyOne without writing a parser, and what happens when the version doesn’t match what you expect. Robot learning needs demonstration data the way language models need text, but until recently every lab and vendor recorded it differently: proprietary formats per robot, inconsistent schemas, no shared way to index or search across datasets. Hugging Face built LeRobotDataset to fix that: one standardized format (Parquet for state and action, MP4 for camera feeds) across radically different embodiments, from SO-100 arms to bimanual ALOHA rigs to industrial Kuka arms. The standardization paid off measurably. SmolVLA, a 450M-parameter policy, was pretrained only on community-uploaded lerobot-tagged datasets on the Hub, and that alone lifted SO-100 task success from 51.7% to 78.3% (source). The format itself had to evolve under that success: the original v2.x layout stored one Parquet file and one MP4 per episode, which started hitting filesystem limits once community datasets reached millions of episodes. lerobot v0.4.0 shipped v3.0 in response, packing many episodes into shared chunked files and supporting streaming without a full download first. FiftyOne ships fo.types.LeRobotDataset as a first-class import path, built around one idea: a sample is an episode, not a file. The fiftyone-lerobot-import agent skill wraps that native support with the parts a script usually skips: detecting whether a Hub dataset is still on v2.x, running the documented conversion before import, and refusing to create a dataset or install a package without asking first. Key takeaways - The fiftyone-lerobot-import agent skill imports LeRobotDataset v3.0 recordings into FiftyOne, where one sample is one episode rather than one file. - The skill auto-detects LeRobot's older v2.x layout (meta/episodes.jsonl, meta/tasks.jsonl) and runs the documented conversion to v3.0 before importing, instead of failing on an unsupported-version error. - Two explicit stop gates hold the run, one before installing any package (pyarrow, huggingface_hub, lerobot) and one before creating the FiftyOne dataset. - Tested end to end against lerobot/aloha_sim_insertion_human, a real v2.1-tagged Hub dataset, which converted to v3.0 and imported as 50 episodes of a bimanual ALOHA arm with zero episodes skipped. - The skill is a thin layer over the open-source FiftyOne SDK (fo.Dataset.from_dir, dataset.add_dir), so the same import runs identically from a chat prompt or a checked-in script. Part one: what is an agent skill, actually? Strip away the buzzwords and a skill is a folder. Inside it is a markdown file that tells an AI coding agent, step by step, how an expert would do a specific job: which files to check first, which command to run, which traps to avoid, and how to tell success from failure. The agent reads that folder like a new hire reading a training guide, then does the work itself in your terminal, on your data. Voxel51 packages this kind of runbook for FiftyOne into an open-source repository, installed with one line: curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills The installer clones the repo, discovers every skill inside it, and lets you pick which ones your agent should have access to going forward. This post is about fiftyone-lerobot-import. Point it at a local LeRobot directory or a Hugging Face Hub repo id, and it detects the version, converts if needed, and builds the FiftyOne dataset. That’s a different problem from the one fiftyone-dataset-import’s MCAP path solves. MCAP’s question is a schema scan: what kind of sensor data is this? LeRobot’s question is about versioning: what version of this format is this, and does it need conversion first? Part Two: A Real LeRobot Dataset, End to End A skill only proves itself on data that actually exercises its hardest path. A dataset already sitting on v3.0 would import cleanly and tell you nothing about the conversion logic, so we looked for a dataset still tagged v2.1 on the Hub. The dataset: lerobot/aloha_sim_insertion_human lerobot/aloha_sim_insertion_human is one of LeRobot’s official example repos: a bimanual ALOHA arm running a simulated peg-insertion task, one task label (“Insert the peg into the socket.”), 50 episodes, 50 fps, 25,000 total frames. | Detail | Value | | --- | --- | | Repo | lerobot/aloha_sim_insertion_human | | Revision pulled | v2.1 | | Episodes | 50 | | FPS | 50 | | robot_type | aloha | | Camera | observation.images.top (video, av1 codec, 480×640) | | observation.state / action dims | 14 / 14 (bimanual: waist, shoulder, elbow, forearm roll, wrist angle, wrist rotate, gripper, same layout on both arms) | | Task | “Insert the peg into the socket.” | What lerobot/aloha_sim_insertion_human contains, pulled at the v2.1 tag. We pulled the Hub’s v2.1 tag specifically, not main (already v3.0), so the conversion path would get exercised for real instead of simulated. The prompt and the version-detection moment The prompt to the agent was one sentence: import lerobot/aloha_sim_insertion_human into FiftyOne. The skill’s inspection step reads meta/info.json and finds codebase_version: v2.1, along with layout markers like meta/episodes.jsonl and meta/tasks.jsonl. Instead of failing on unsupported versions, the skill detects the older format, hits the packages stop gate, and runs the conversion script (python-m lerobot. scripts. convert_dataset_v21_to_v30) to update the dataset to v3.0, ensuring reliable import even with legacy data. 1. Hits the packages stop gate and asks before installing lerobot (needed only for the conversion). 1. Runs the conversion: python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id lerobot/aloha_sim_insertion_human --root <path> --push-to-hub=false. 1. Keep it honest. The installed lerobot release (v0.6.2) didn’t expose the converter at the documented lerobot.datasets.v30.* module path; it was still at the older lerobot.scripts.* path. The skill’s own troubleshooting notes already anticipate this exact fallback, and it’s what happened on the first real run, not a hypothetical edge case. 1. Re-checks meta/info.json, confirms codebase_version: v3.0, and the new layout: meta/episodes/chunk-000, meta/tasks.parquet, meta/stats.json. 1. Hits the creation stop gate and presents the full import plan before touching anything: LeRobot Import Plan for lerobot/aloha_sim_insertion_human: Source: - codebase_version: v3.0 (converted from v2.1: yes) - 50 episodes, 50 fps, robot_type: aloha - camera: observation.images.top (video, av1) - observation.state: 14 dims, action: 14 dims - tasks: 1 ("Insert the peg into the socket.") Import: - Dataset name: lerobot-aloha-sim-insertion-human - Episodes: all - Persistent: yes Proceed with import? (yes/no) The payoff: 50 episodes, zero skipped dataset.media_type comes back "multimodal". All 50 episodes import; dataset.info["lerobot"]["skipped_episodes"] is an empty list: a clean import, worth stating plainly rather than manufacturing a gap that isn’t there. Every episode runs exactly 10 seconds (dataset.bounds("duration") returns (10.0, 10.0)), which checks out at 50 fps × 500 frames/episode. Opening episode 0 in the App shows the observation.images.top tile: two robot arms working over a blue and red object, with the episode header reading “Insert the peg into the socket. · 10s.” That’s what “one sample equals one episode” looks like in the viewer, not an abstraction: a single click surfaces the whole 10-second demonstration, not one video frame. Part Three: The Same LeRobot Import, in Code A chat prompt is a good way to explore a dataset you haven’t seen before, but a pipeline that runs on every new upload needs code, not a conversation. Everything the skill did above sits on top of the open-source FiftyOne SDK. Here’s the full pipeline, runnable from a fresh environment against the same Hub dataset used in Part Two. Step 1: Install the dependencies pip install "pyarrow>=10.0.0" huggingface_hub lerobot lerobot is only needed for the v2.1 → v3.0 conversion below. A dataset already on codebase_version: v3.0 only needs pyarrow and huggingface_hub. Step 2: Download from the Hub fo.Dataset.from_dir reads from a local directory, not directly from the Hub, so pull the files down first: from huggingface_hub import snapshot_download local_dir = snapshot_download( repo_id="lerobot/aloha_sim_insertion_human", repo_type="dataset", revision="v2.1", local_dir="/abs/path/lerobot/aloha_sim_insertion_human", ) Step 3: Convert v2.1 to v3.0 python -m lerobot.scripts.convert_dataset_v21_to_v30 \ --repo-id lerobot/aloha_sim_insertion_human \ --root /abs/path/lerobot/aloha_sim_insertion_human \ --push-to-hub=false Skip this step for any dataset already tagged v3.0 on the Hub; for example, pull revision="v3.0" in Step 2 instead. Step 4: Import into FiftyOne import fiftyone as fo dataset = fo.Dataset.from_dir( dataset_dir=local_dir, dataset_type=fo.types.LeRobotDataset, name="lerobot-aloha-sim-insertion-human", persistent=True, ) assert dataset.media_type == "multimodal" fo.launch_app(dataset) Adding a second recording session to the same dataset is the same four steps repeated against the new source directory, then dataset.add_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset) instead of from_dir. No Parquet reader, no video decode loop, no custom viewer. The importer reads meta/info.json and meta/episodes/*.parquet, nothing in the video files, so it stays fast even on large sources. The skill’s stop gates are a conversation; the SDK calls underneath are the same ones either way. Where This Leaves You Every team re-solves “is my data in the format my tools expect” by hand, one script at a time, and LeRobot is just the latest format to need that check. fiftyone-lerobot-import turns that check into something an agent already knows how to run correctly against your files, every time. - Dataset: lerobot/aloha_sim_insertion_human, part of LeRobot’s official example datasets, Hugging Face - Skill repo: voxel51/fiftyone-skills, installed the same one-line way as Part One Frequently Asked Questions How do you import a LeRobot dataset into FiftyOne? Two ways: ask an agent with the fiftyone-lerobot-import skill, or call fo.Dataset.from_dir(..., dataset_type=fo.types.LeRobotDataset) directly. Both require the dataset to be on codebase_version: v3.0. What’s the difference between LeRobot v2.x and v3.0? v2.x stores one Parquet file and one MP4 per episode. v3.0 packs many episodes into shared chunked files and supports streaming. fo.types.LeRobotDataset only supports v3.0. v2.x sources need conversion first. Do I need the lerobot package to import into FiftyOne? Only for converting a v2.x source to v3.0. A straight v3.0 import needs only pyarrow>=10.0.0. How does FiftyOne represent a LeRobot dataset? One sample per episode, not per file, with media_type == "multimodal" and fields like episode_index, task, duration, and fps. What happens if some episodes in the source are malformed? They’re reported under dataset.info["lerobot"]["skipped_episodes"] instead of being silently dropped or silently accepted.

10 Multimodal MCAP Datasets in 2026: SLAM Benchmarks, Sensor Fusion, and Ground-Truth Object Detection in FiftyOne A field guide to 10 robotics datasets (driving, aquatic, forest, indoor, and synthetic) repackaged as time-synchronized MCAP recordings for FiftyOne’s native multimodal viewer, with the exact sensor suite, label situation, and a direct link to the paper, the dataset, and (where one exists) a live FiftyOne Space for each. A camera on a driving rig fires at 10-20 Hz, its LiDAR spins at 10 Hz, its inertial measurement unit (IMU) ticks at 200-400 Hz, and its GPS/INS updates entirely on its own clock. None of these streams line up into a clean “one image = one label = one sample” unit, and yet that unit is what almost every computer vision tool assumes you have. Robot data doesn’t work that way. A single recording is a bundle of asynchronous, multi-rate streams captured over minutes, not a folder of independent frames. Trying to force it into per-frame samples either throws away the temporal relationship between sensors or multiplies your sample count by however many topics you’re logging. That’s the problem MCAP was built to solve, and it’s why FiftyOne added native multimodal support for it. We pulled together 10 datasets, all repackaged as MCAP episodes on the Voxel51 Hugging Face org, spanning urban driving, handheld SLAM, aquatic navigation, off-road forest robotics, indoor human-robot interaction, and one fully synthetic set. Below: what MCAP actually is, the data curation problems that show up over and over when repackaging real robot logs into it, a full comparison and schema reference, and a domain-by-domain tour of all 10 datasets with links to the paper, the dataset, and the FiftyOne Space where one exists. Key Takeaways - 10 datasets, 89 total episodes, spanning ground-vehicle driving, campus/handheld SLAM, aquatic navigation, off-road forest robotics, indoor social navigation, and one synthetic set (TartanGround). Every one is a FiftyOne multimodal dataset, where each sample is a continuous recording, not a single frame. - Only 3 of 10 carry real, embedded object-level labels: SemanticSpray++ (all 36 episodes fully labeled, 2D camera boxes, 3D LiDAR boxes, LiDAR + radar point-wise semantics), boreas-multimodal (only 6 of its 8 episodes, the boreas-objects-v1 windows, not the 2 driving episodes), and cmht-autonomous-driving (3D + derived 2D detections on all 4 drives). 6 more (yuto-mms-multimodal, oxford-spires-multimodal, canoe-multimodal, fomo-multimodal-sample, treescope-vat0723-multimodal, TartanGround) are raw SLAM/odometry sensor data with zero object annotations. - navwareset sits in between. Pedestrian 3D cuboids exist and are real, but they’re logged as foxglove.SceneUpdate entities inside the MCAP timeline, not a FiftyOne label field, the same modeling choice used for Boreas’s and CMHT’s boxes, since a multi-minute episode has no single fixed-length list a sample-level label field could hold. - Sensor suites go well beyond camera + LiDAR: boreas-multimodal, canoe-multimodal, and fomo-multimodal-sample all carry radar; canoe-multimodal adds imaging sonar; cmht-autonomous-driving adds a thermal/IR camera; fomo-multimodal-sample adds stereo audio. Three datasets (yuto-mms-multimodal, oxford-spires-multimodal, boreas-multimodal) ship survey-grade ground-truth trajectories accurate to 1-4 cm. - Licensing is genuinely mixed: CC BY 4.0 is most common, but oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA (non-commercial), cmht-autonomous-driving is CC0, boreas-multimodal’s license is unlisted/unknown, and yuto-mms-multimodal has a genuine conflict: the project site says CC BY-NC-SA 4.0, the live Zenodo records serve CC BY 4.0. - 4 of 10 have no arXiv preprint at all (yuto-mms-multimodal, navwareset, canoe-multimodal, cmht-autonomous-driving), DOI/journal citation only, so don’t go hunting for a paper ID that doesn’t exist. What is MCAP, and why did it become the physical AI format? MCAP (pronounced “em-cap”) is a modular, serialization-agnostic binary container format for timestamped pub/sub data. A single .mcap file holds any number of named channels (camera frames, LiDAR scans, IMU samples, GPS fixes, transform trees), each tagged with its own schema, all indexed together so a reader can seek to any timestamp without scanning the whole file. Foxglove built it in 2022 after noticing that robotics teams kept burning engineering time reinventing custom binary formats, only to make third-party tooling harder later. It’s now the default log format for ROS 2 (since the Iron release in May 2023), replacing the SQLite3-backed .db3 bags, which struggled with high-throughput sequential writes. The properties that make it a good robotics log format are the same ones that make it a good physical AI training-data format: - Serialization-agnostic. One file can hold ROS 1 messages, ROS 2 messages, Protobuf, JSON Schema, and FlatBuffers side by side. That’s exactly the mix you get on a real rig with a ROS-based LiDAR driver, a vendor SDK camera, and a custom logger. - Self-contained. Message schemas are embedded in the file itself, so an MCAP recorded years ago is still readable without the original robot software stack, a real concern for datasets that are already 1-5 years old. - Append-only writing. Data streams to disk without ever seeking backward, so a recording survives a software crash or power loss mid-capture rather than being corrupted. - Chunk-indexed for fast seeking. A chunk index at the end of the file lets a reader jump directly to a timestamp instead of scanning from the start. That’s the specific mechanic that makes scrubbing a 20-minute episode in FiftyOne’s multimodal viewer responsive instead of painful. - Built-in compression (LZ4 or Zstandard) at the chunk level, without sacrificing indexed random access (mcap.dev; Segments.ai’s comparison; Voxel51’s MCAP glossary entry). Common data curation issues when repackaging real robot logs into MCAP Each of the 10 datasets below started life as raw ROS bags, per-file sensor dumps, or vendor SDK output and had to be converted. The same handful of problems showed up repeatedly across completely unrelated source datasets, worth knowing before you repackage your own robot logs. - No native MCAP schema for polar or non-camera sensors. Radar (boreas-multimodal, canoe-multimodal, fomo-multimodal-sample) and imaging sonar (canoe-multimodal) have no dedicated Foxglove schema. All three decode their radar into a bird’s-eye-view foxglove.RawImage pseudo-image rather than a real camera view, the closest available representation, not a native one. - Broken or placeholder timestamps that require rebasing; not trusting. treescope-vat0723-multimodal’s processed bag stamped every message with the wall-clock time of an offline reprocessing job run weeks after the actual flight, not the flight’s capture time. Left uncorrected, an episode would nominally span that entire gap with real data in only two tiny slivers. cmht-autonomous-driving’s LiDAR and camera topics had a placeholder sec=0 header on every raw message. navwareset’s robot /tf carried real but non-physical roll/pitch/z noise, verified against the same bag’s adaptive Monte Carlo localization (AMCL) pose (which was exactly zero on every message) before flattening it. - Frame-count assumptions that quietly break on real data. cmht-autonomous-driving’s labels are published against a separately numbered “extracted frame” release, and an initial assumption of “labeled frame N = the Nth bag message” held for exactly 1 of 4 drives. The other 3 had extra bag messages that the extraction dropped, growing throughout the drive. The fix required content-based matching (pixel equality for images, near-exact point-coordinate equality for LiDAR), not positional counting. - Missing calibration that has to be derived, never fabricated. semanticspray-plusplus ships with zero camera calibration anywhere in the source release, the devkit, or the community-supplied calibration file, so its camera stream is intentionally left unconnected to the transform tree rather than given an invented pose. oxford-spires-multimodal’s depth/normal-map pipeline needs open3d, which has no wheel for the build machine’s platform, so the devkit’s own math was reimplemented in NumPy/OpenCV instead of skipped. - Placeholder or leaked values that look like real per-point data but aren’t. canoe-multimodal’s LiDAR preallocates a 131,072-point array per frame; 70-93% of those points are (0,0,0) “no return” filler that has to be filtered out, or the 3D tile shows a dense blob at the origin. semanticspray-plusplus’s radar z-column is a dataset-wide constant that’s exactly the sensor’s calibrated mount height, leaked from upstream extraction rather than a real per-point measurement, so height is set via a static transform instead. Which MCAP dataset should you use? | Dataset | Domain | Episodes | Sensor Suite | Object Labels? | License | | --- | --- | --- | --- | --- | --- | | YUTO MMS | Ground-vehicle mobile mapping (SLAM) | 2 | Panoramic camera (Ladybug 5), tilted 32-beam LiDAR, GNSS/INS, IMU | None | Conflicting (site: CC BY-NC-SA; Zenodo: CC BY) | | Oxford Spires | Handheld backpack SLAM/reconstruction | 6 | 3× fisheye camera, 64-beam LiDAR, IMU, derived depth/normal maps | None | CC BY-NC-SA 4.0 | | CANOE | Aquatic uncrewed surface vessel (USV) navigation | 4 | Stereo camera, 360° radar, imaging sonar, 128-beam LiDAR, GNSS/INS, IMU | None | CC BY 4.0 | | FoMo | Off-road forest uncrewed ground vehicle (UGV) navigation | 6 | Stereo + mono camera, 2× LiDAR, FMCW radar, stereo audio, 2× IMU, post-processed kinematic (PPK)-GNSS | None | CC BY 4.0 | | TreeScope (VAT-0723) | uncrewed aerial vehicle (UAV) forestry LiDAR mapping | 10 | Stereo camera, 64-beam LiDAR, GPS, IMU | None | CC BY-NC-SA 4.0 | | TartanGround | Synthetic ground-robot navigation | 6 | Camera, segmentation stream, LiDAR, ego-pose, IMU | None | CC BY 4.0 | | SemanticSpray++ | Adverse-weather driving (vehicle spray) | 36 | Camera, 3× LiDAR, radar | 2D boxes, 3D boxes, LiDAR + radar point semantics (all 36 episodes) | CC BY 4.0 | | Boreas / Boreas-RT | Multi-season driving | 8 | Camera, 128-beam LiDAR (+ FMCW Aeva on Boreas-RT), 360° radar, GNSS/INS, IMU | 3D boxes (6 of 8 episodes only) | Unknown | | CMHT | Urban driving, sensor fusion | 4 | Camera, thermal/IR camera, LiDAR, mmWave radar, GPS/IMU | 3D boxes + derived 2D boxes (all 4 episodes) | CC0 1.0 | | NavWareSet | Indoor social navigation human-robot interaction (HRI) | 7 | Robot + external station: LiDAR ×2, camera ×2, AMCL pose, cmd_vel | Pedestrian 3D cuboids (SceneUpdate, not a label field) | CC BY 4.0 | All 10 MCAP datasets side by side: domain, episode count, sensor suite, whether real object labels are embedded, and license. Keep it honest. “Episodes” reflects each FiftyOne build’s current scope, not the full source dataset. Every one of these is a partial repackaging (see each domain section for exactly how much of the source is covered and why). What does "episode" mean in FiftyOne? For every dataset below, dataset.media_type == "multimodal" and one sample is one episode, one continuous recording, stored as a single .mcap file. There’s no per-frame image or point-cloud sample sitting alongside it; every camera frame, LiDAR scan, IMU sample, and transform inside that episode is decoded live by FiftyOne’s multimodal viewer, which renders Image, 3D, Map, Plot, Logs, and Message tiles in sync as you scrub through the recording: import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/boreas-multimodal") session = fo.launch_app(dataset) Object-level labels, where they exist, follow the same logic: a 3D box that’s only valid for one instant inside a 10-minute recording can’t live in a fixed-length sample field, so it’s logged as another timestamped channel inside the same MCAP (foxglove.SceneUpdate for 3D cuboids, foxglove.ImageAnnotations for 2D overlays), decoded in sync with the sensor data rather than stored separately. Dataset schema reference: media types, sample fields, and links | Dataset | Media Type | Notable Sample Fields | Label Fields | | --- | --- | --- | --- | | YUTO MMS | multimodal | sequence_id, route_length_m, collection_site, has_gps/has_imu/has_pointcloud | None | | Oxford Spires | multimodal | location, recording_date, has_rgb_lidar, has_depth, has_surface_normals, num_cameras | None | | CANOE | multimodal | route, radar_max_range_m, has_gps, has_imu (False, generic JSON, not schema-decoded) | None | | FoMo | multimodal | trajectory, deployment_date, gt_origin_utm, n_robosense_scans/n_leishen_scans, has_audio | None | | TreeScope | multimodal | site, session_id, platform, lidar_model, tags (manual_flight/autonomous_flight) | None | | TartanGround | multimodal | environment, trajectory, num_frames, duration | None | | SemanticSpray++ | multimodal | vehicle_type, ego_velocity_kmh, distance_to_object_m, has_object_labels (True for all 36) | 2D boxes, 3D boxes, point-wise semantics (native MCAP schemas, not FiftyOne label fields) | | Boreas / Boreas-RT | multimodal | episode_type (odometry/object_detection), dataset_variant, has_boxes, has_aeva | 3D boxes on 6 of 8 episodes | | CMHT | multimodal | drive_name, weather, lighting, is_preliminary_sync, has_3d_labels, has_2d_labels | 3D + derived 2D detections, all 4 | | NavWareSet | multimodal | scenario, robot_platform, behavior (social/non-social), has_pedestrian_annotations | Pedestrian cuboids (SceneUpdate) | What each dataset looks like once loaded in FiftyOne: media type, the sample fields you can filter on, and where labels actually live. Links: papers, datasets, and FiftyOne Spaces YUTO MMS - YUTO MMS: A comprehensive SLAM dataset for urban mobile mapping with tilted LiDAR and panoramic camera integration - YUTO MMS Hugging Face Dataset - YUTO MMS FiftyOne Space Oxford Spires - Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods - Oxford Spires Hugging Face Dataset - Oxford Spires FiftyOne Space CANOE - CANOE Hugging Face Dataset - CANOE FiftyOne Space FoMo - FoMo: A Multi-Season Dataset for Robot Navigation in Forêt Montmorency - FoMo Hugging Face Dataset - YUTO MMS FiftyOne Space TreeScope (VAT-0723) - TreeScope: An Agricultural Robotics Dataset for LiDAR-Based Mapping of Trees in Forests and Orchards - TreeScope Hugging Face Dataset - TreeScope FiftyOne Space TartanGround - TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation - TartanGround Hugging Face Dataset SemanticSpray++ - SemanticSpray++: A Multimodal Dataset for Autonomous Driving in Wet Surface Conditions - Energy-based Detection of Adverse Weather Effects in LiDAR Data - SemanticSpray++ Hugging Face Dataset - SemanticSpray++ FiftyOne Space Boreas / Boreas-RT CMHT - Boreas: A Multi-Season Autonomous Driving Dataset - Boreas Road Trip: A Multi-Sensor Autonomous Driving Dataset on Challenging Roads - Boreas Hugging Face Dataset - Boreas FiftyOne Space CMHT - CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving - CMHT Hugging Face Dataset - CMHT MMS FiftyOne Space NavWareSet - NavWareSet: A dataset of socially compliant and non-compliant robot navigation - NavWareSet Hugging Face Dataset Ground-vehicle and campus SLAM datasets (raw, no object labels) Both of these are raw multi-sensor recordings built for SLAM/odometry/reconstruction benchmarking, not perception. Neither carries a single object-level annotation. Their “ground truth” is a survey-grade trajectory, not a bounding box. YUTO MMS !YUTO MMS MCAP episode in FiftyOne showing an accumulated RGB-colorized LiDAR map, a 360-degree Ladybug panorama, and the GPS track around the Teledyne Optech campus. YUTO MMS MCAP episode in FiftyOne showing an accumulated RGB-colorized LiDAR map, a 360-degree Ladybug panorama, and the GPS track around the Teledyne Optech campus. YUTO MMS (York University Teledyne Optech Mobile Mapping System) is a SLAM benchmark collected with Teledyne Optech’s Maverick mobile mapping unit: a 360° Ladybug 5 panoramic camera (six stitched CCD sensors, 8000×4000), a Velodyne HDL-32E LiDAR tilted 45° off the camera axis, and a NovAtel GNSS/INS, recorded across a controlled parking-lot sequence in Vaughan, Ontario and a residential-loop sequence around York University’s Keele Campus in Toronto. This build covers 2 of the source’s 4 sequences (A and D); B and C are still in progress, since each requires re-tuning the accumulated-map voxel size and update rate per sequence rather than reusing a previous one’s settings. Every LiDAR point is colorized with real RGB sampled from the nearest-in-time panorama via the dataset’s own boresight calibration, not a synthetic colormap. Fields. sequence_id, route_length_m, collection_site, duration_s, plus has_image/has_pointcloud/has_gps/has_imu capability flags. No label fields exist, and none should. The source paper’s entire “Dataset” section covers only raw sensor streams and a photogrammetric ground-truth trajectory. Keep it honest. The license is genuinely conflicting: the project site and README both state CC BY-NC-SA 4.0, but every live Zenodo record for every sequence checked directly serves CC BY 4.0 via its own rel="license" header. Verify current status before redistributing. Dataset · FiftyOne Space Oxford Spires !FiftyOne multimodal viewer playing an Oxford Spires MCAP episode, with RGB-colorized LiDAR of the Bodleian Library and three fisheye camera frustums in the 3D tile. FiftyOne multimodal viewer playing an Oxford Spires MCAP episode, with RGB-colorized LiDAR of the Bodleian Library and three fisheye camera frustums in the 3D tile. Oxford Spires is the Oxford Robotics Institute’s large-scale LiDAR-visual localization, reconstruction, and radiance-field benchmark, captured by walking a handheld backpack unit called Frontier: three synchronized fisheye cameras (forward/left/right), a 64-beam LiDAR, and an IMU, through six historic Oxford landmarks (Bodleian Library, Christ Church, Keble College, New College, Blenheim Palace, Radcliffe Observatory Quarter). Each site also has a millimeter-accurate Terrestrial LiDAR Scanner reference model, used to produce centimeter-accurate ground-truth trajectories via iterative closest point (ICP) registration. Six of the source’s 24 sequences make it into this build, one per site, choosing the smallest available sequence at each landmark. Alongside the raw streams, every episode also carries the source devkit’s own derived depth maps, HSV depth overlays, and surface-normal maps per camera, so the devkit’s canonical visualization is reproducible directly in the App. Fields. location, recording_date, run_number, plus has_rgb_lidar/has_depth/has_surface_normals/num_cameras capability flags. No object-level labels. The source dataset’s “ground truth” is millimeter-accurate TLS 3D models and centimeter-accurate trajectories, not per-sample annotations. Good to know. Why is the Map tile empty for every episode? There’s no GNSS/GPS stream anywhere in the source dataset, so the App’s Map tile has nothing to show, by design, not a parsing gap. Dataset · FiftyOne Space Aquatic and off-road robotics datasets (raw, no object labels) Same story as the ground-SLAM group: raw sensor streams plus a post-processed ground-truth pose, no object annotations. But the environments couldn’t be more different: a lake, a boreal forest, and a forest canopy from above. CANOE !Height-colored LiDAR point cloud of a lake shoreline and tree line from a CANOE MCAP episode in FiftyOne's 3D tile. Height-colored LiDAR point cloud of a lake shoreline and tree line from a CANOE MCAP episode in FiftyOne's 3D tile. CANOE (Canadian Aquatic Navigation for Observation of the Environment) is ASRL’s marine navigation dataset, collected on an uncrewed surface vessel traversing a lake and reservoir in Ontario over three days. The sensor suite is genuinely unusual for a robotics dataset: a 360° Navtech scanning radar (dual range modes: 1000 m long-range or 150 m Doppler), a 128-beam Ouster LiDAR, a stereo camera, an Oculus imaging sonar, dual Starfish sidescan sonar, and a NovAtel GNSS/INS post-processed to 2-4 cm RMS accuracy. Four of the source’s 8 public sequences are included here, chosen for sensor/environment diversity: three fixed-duration windows plus the officially curated “Sample” sequence in full. Fields. route, radar_max_range_m, source, plus has_image/has_pointcloud/has_gps capability flags. has_imu is False for every episode in this build even though an /imu topic exists, because it rides on a generic JSON channel rather than a schema-decoded one. Keep it honest. Sidescan sonar is out of scope for this build. sidescan.xtf/sidescan.csv aggregate an entire sequence’s pings into single files with no per-frame breakdown, so they were never decoded into any MCAP channel. Dataset · FiftyOne Space FoMo !FoMo MCAP episode in FiftyOne showing the Navtech radar bird's-eye-view stream next to a colorized forest LiDAR point cloud. FoMo MCAP episode in FiftyOne showing the Navtech radar bird's-eye-view stream next to a colorized forest LiDAR point cloud. FoMo is a year-long robotic data collection recorded in Forêt Montmorency, a boreal forest 80 km north of Quebec City, using a Clearpath Warthog UGV carrying two LiDARs (RoboSense + Leishen), an frequency-modulated continuous wave (FMCW) radar, a stereo + mono camera pair, two IMUs, stereo microphones, and PPK-GNSS ground truth. The full source is 9.43 TB across 60 sessions; this build samples 6 episodes, one per trajectory type, across 6 deployments spanning winter (-19°C, over a meter of snow) through summer conditions, kept under a 30 GB budget. Fields. trajectory, trajectory_description, deployment_date, gt_origin_utm, sensor-specific frame counts (n_robosense_scans, n_leishen_scans, n_navtech_frames, n_audio_left_clips), plus has_image/has_pointcloud/has_imu/has_gt_pose/has_audio flags. Keep it honest. This is a 6-of-60-session recon sample under a size budget, not a statistically representative subset. Don’t use it as a stand-in for the full dataset’s scale in a SLAM/odometry benchmark. Dataset · FiftyOne Space TreeScope (VAT-0723) !TreeScope VAT-0723 MCAP episode in FiftyOne with UAV LiDAR point clouds, two stereo camera tiles, an odometry plot, and the Map tile. TreeScope VAT-0723 MCAP episode in FiftyOne with UAV LiDAR point clouds, two stereo camera tiles, an odometry plot, and the Map tile. TreeScope is a LiDAR dataset for precision agriculture and forestry, collected across six forest and orchard sites with UAV and mobile-robot platforms. This build covers 10 episodes from just one site, VAT-0723 (Appomattox-Buckingham State Forest, Virginia), the only confirmed TreeScope site with released GPS data, making FiftyOne’s Map tile achievable at all. The UAV platform (a 4.2 kg Falcon 4) carries an Ouster OS1-64 LiDAR, stereo cameras, a VectorNav IMU, and a UBlox GPS. Each episode merges that session’s raw and Faster-LIO-processed ROS bag pair, remapping the processed trajectory to /refined/Odometry so both the onboard and offline-refined estimates stay distinguishable. Fields. site, session_id, platform, lidar_model, flight-mode tags (manual_flight/autonomous_flight), plus has_image/has_pointcloud/has_gps/has_imu/has_logs flags. Keep it honest. No ground truth is attached to any of these 10 episodes, deliberately. TreeScope’s site-level semantic-segmentation labels exist but their labeled-frame timestamps fall entirely outside every chosen episode’s flight window. metadata/VAT-0723.json independently confirms no semantic labels or diameter at breast height (DBH) measurements for these sessions. This is also a non-commercial dataset (CC BY-NC-SA 4.0). Dataset Synthetic ground-robot navigation TartanGround !TartanGround synthetic MCAP episode in FiftyOne showing a night city camera stream, its paired segmentation stream, and LiDAR playing in sync. TartanGround synthetic MCAP episode in FiftyOne showing a night city camera stream, its paired segmentation stream, and LiDAR playing in sync. TartanGround is CMU AirLab’s large-scale synthetic dataset for ground-robot perception and navigation, generated across a range of simulated environments. This build covers 6 trajectories, one per environment (AbandonedFactory, CyberPunkDowntown, GreatMarsh, Hospital, JapaneseCity, NordicHarbor), each converted to a native multimodal MCAP episode carrying a front camera, its paired segmentation stream, per-frame LiDAR point clouds, ego pose, and IMU telemetry on a shared 10 Hz frame clock. Fields. environment, trajectory, num_frames, duration. Streams: /front-camera (JPEG), /front-segmentation (PNG), /lidar, /ego-pose, /imu.plot. Good to know. Unlike every other raw dataset in this post, TartanGround does ship a segmentation stream (/front-segmentation), but like the object-detection boxes elsewhere, it’s a synchronized MCAP channel decoded live in the Image tile, not a FiftyOne Segmentation label field. Dataset Datasets with real embedded object detections These three are the exception to everything above: real, dense, per-frame object annotations, logged as additional timestamped channels inside the same MCAP timeline as the sensor data, decoded live by the multimodal viewer’s Image and 3D tiles in sync with playback, rather than stored as separate FiftyOne label fields. SemanticSpray++ !SemanticSpray++ MCAP episode in FiftyOne with a 3D LiDAR box on the lead van and point-wise spray semantics colorized in the point cloud. SemanticSpray++ MCAP episode in FiftyOne with a 3D LiDAR box on the lead van and point-wise spray semantics colorized in the point cloud. SemanticSpray++ labels a 36-scene subset of the RoadSpray dataset: an ego vehicle following a lead vehicle (a small car or a large van) on wet pavement at matched speeds from 50-130 km/h, generating a trailing water-spray plume off the wet surface. Every one of the 36 episodes in this build has the complete label set: 2D camera boxes, 3D LiDAR boxes, LiDAR point-wise semantics (background/foreground/noise), and radar point-wise semantics, all for the lead vehicle. Across the 36 episodes: 1,362 Car and 1,225 Van 2D/3D box instances, plus 77.8 million labeled LiDAR points. Fields. vehicle_type (Golf/Crafter), ego_velocity_kmh, distance_to_object_m, has_object_labels (True for all 36). Label channels: /camera/annotations (2D boxes), /objects/lidar_boxes (3D boxes), colorized /velodyne_points (point semantics), colorized radar points. Keep it honest. The other 171 scenes in the same source download (the earlier, LiDAR-only “SemanticSpray” release with no boxes) are deliberately not included. This build targets exactly the 36-scene, fully-labeled SemanticSpray++ subset. Dataset · FiftyOne Space Boreas / Boreas-RT !Boreas MCAP episode in FiftyOne showing 3D object cuboids over the LiDAR scene, with synchronized camera and radar tiles below. Boreas MCAP episode in FiftyOne showing 3D object cuboids over the LiDAR scene, with synchronized camera and radar tiles below. Boreas is ASRL’s multi-season driving dataset, collected by repeatedly driving a route near the University of Toronto over a full year: sun, rain, falling snow. Boreas Road Trip (Boreas-RT) extends it with 9 new, more diverse routes and adds an Aeva FMCW Doppler LiDAR plus a standalone IMU. A separate dedicated recording, boreas-objects-v1, provides 326,180 3D object annotations (Car/Pedestrian/Cyclist/Misc) across 7,111 labeled LiDAR frames. This build covers 2 driving episodes (one Boreas, one Boreas-RT, each a 10-second window) plus 6 object-detection windows carved from boreas-objects-v1. Fields. episode_type (odometry/object_detection), dataset_variant, has_boxes, has_aeva, has_dmu_imu. Label channels (object-detection episodes only): /boxes (3D SceneUpdate cuboids), /camera/boxes and /radar/boxes (2D wireframe overlays), all carrying label/uuid/numPoints metadata inspectable via FiftyOne’s click-to-inspect panel. Keep it honest. has_boxes is True for only 6 of the 8 episodes. The 2 driving/odometry episodes carry no object annotations at all. Only obj_train-split windows are included; obj_test ground truth is withheld by the source dataset for leaderboard fairness. Dataset · FiftyOne Space CMHT !CMHT night-rain MCAP episode in FiftyOne with car and truck boxes on the RGB camera, the same detections on the thermal IR stream, and 3D cuboids in the LiDAR scene. CMHT night-rain MCAP episode in FiftyOne with car and truck boxes on the RGB camera, the same detections on the thermal IR stream, and 3D cuboids in the LiDAR scene. The CMHT Autonomous Dataset is McMaster University’s multi-sensor driving dataset recorded around Hamilton, Ontario: a single rig combining a Velodyne LiDAR, an mmWave radar, a monocular RGB camera, and a FLIR thermal/IR camera, across four drives spanning dusk/clear and night/rain conditions. Over 9,000 frames are labeled with 3D position, size, rotation, class (Car/Truck/Van/Pedestrian/Bus/LongVehicle), and a persistent object ID. This repackaging re-embeds those separately-published labels back into each drive’s MCAP timeline, synced to the exact sensor message each was originally annotated from. Fields. drive_name, weather, lighting, is_preliminary_sync, has_3d_labels, has_2d_labels. Label channels: /Labels_3D (vision_msgs/Detection3DArray, one message per labeled LiDAR frame) and /Labels_2D_front//Labels_2D_ir (vision_msgs/Detection2DArray). Keep it honest. The 2D boxes are derived, not manually annotated. Only the 3D LiDAR-frame boxes were hand-labeled (via SUSTechPoints); the 2D camera and thermal boxes are projections of those same 3D boxes onto each image plane, computed by this repackaging using the authors’ own calibration and projection method. is_preliminary_sync is also True for 3 of the 4 drives: only the LiDAR triggers the cameras, while radar, GPS, and IMU are approximately, not frame-exactly, time-aligned. Dataset · FiftyOne Space Social navigation and human-robot interaction NavWareSet !NavWareSet Circular Crossing MCAP episode in FiftyOne showing the Jackal robot's camera view, the ground-station camera, and pedestrian and trajectory annotations in the 3D scene. NavWareSet Circular Crossing MCAP episode in FiftyOne showing the Jackal robot's camera view, the ground-station camera, and pedestrian and trajectory annotations in the 3D scene. NavWareSet is a multimodal dataset of socially-compliant and non-compliant robot navigation around human pedestrians, recorded in a controlled indoor environment across seven canonical social-navigation scenarios (Frontal Approach, Pedestrian Obstruction, Blind Corner, Following Human, Perpendicular Traffic, Circular Crossing, Object Handover). Every scene is captured simultaneously by the robot’s own sensors and by a stationary external Ground-truth Recording Station (a LiDAR mounted above a camera), which also provides the point cloud human trajectories were manually annotated on. Seven of the source’s 48 scenes are included here, Clearpath Jackal platform only. The Toyota HSR platform isn’t imported yet, though the merge pipeline already supports it. Fields. scenario, robot_platform, behavior (social/non-social), participant_group, has_pedestrian_annotations. Label channel: /annotations/pedestrians (foxglove.SceneUpdate, persistent per-person cuboids carrying the source’s own tracking ID). Good to know. Why isn’t there a Detections field for the pedestrian boxes? Pedestrian cuboids are a time-synced channel meant to play back alongside the LiDAR/camera channels in the 3D tile: SceneUpdate entities carry a persistent entity ID across frames, matching the source annotation format’s per-person UUID directly, with no separate tracking field needed. Dataset Where this is all headed: physical AI Across all 10 datasets, the actual training bottleneck is real, timestamped, multi-sensor recordings of something moving through the world (a car, a boat, a UAV, a robot navigating around a person) with enough synchronized modalities to reconstruct what happened. That’s the raw material world models and embodied agents train on, and it’s the shape of data MCAP was built to hold and FiftyOne’s multimodal viewer was built to make explorable without a custom bag-parsing script. Try it yourself pip install -U fiftyone import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/semanticspray-plusplus") # or any dataset above session = fo.launch_app(dataset) - FiftyOne on GitHub - Voxel51 on Hugging Face - MCAP format specification Frequently Asked Questions What is an MCAP file? MCAP is a modular, serialization-agnostic binary container format for timestamped pub/sub data: one file holds any number of named channels (camera, LiDAR, IMU, GPS, transforms), each indexed for fast seeking. It’s been the default log format for ROS 2 since the Iron release. Can I view ROS bag data in FiftyOne? Not directly. ROS 1 .bag and ROS 2 .db3 files need to be converted to .mcap first (via the mcap CLI, rosbags, or a custom script), at which point FiftyOne’s media_type: "multimodal" support decodes them automatically based on the .mcap file extension. Which of these datasets actually have object detection labels? Only three: semanticspray-plusplus (all 36 episodes, 2D boxes, 3D boxes, point-wise semantics), boreas-multimodal (6 of its 8 episodes, via the has_boxes field), and cmht-autonomous-driving (3D boxes plus derived 2D projections, all 4 episodes). navwareset has real pedestrian cuboids but as a SceneUpdate channel, not a FiftyOne label field. The remaining 6 are raw sensor data with no object annotations at all. What’s the difference between a FiftyOne multimodal dataset and a grouped dataset? A multimodal dataset (what every dataset in this post is) has media_type: "multimodal" and one sample per .mcap episode, with every stream inside decoded live by the viewer’s tiles. A grouped dataset instead has media_type: "group" and multiple related samples (e.g., different camera slices, a 3D scene) linked under one group field, a different mechanism used when labels need to attach as ordinary per-frame FiftyOne fields rather than live inside an MCAP timeline. Which MCAP datasets can I use commercially? Check each license individually. semanticspray-plusplus, fomo-multimodal-sample, canoe-multimodal, navwareset, and TartanGround are CC BY 4.0; cmht-autonomous-driving is CC0 1.0. oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA 4.0 (non-commercial only). boreas-multimodal’s license is unlisted. yuto-mms-multimodal has a genuine conflict between its project site (CC BY-NC-SA) and its live Zenodo records (CC BY). Verify directly before relying on either.