fiftyone

How to Find Edited Copies of Images: Image Copy Detection with pHash, CLIP, and FiftyOne Exact-duplicate search misses a mirrored, cropped, or filtered copy of an image, so this post combines a perceptual hash and a CLIP embedding into one rule you control and shows the copies as a graph in FiftyOne. Your model's test metrics look great. Then someone opens the test set and finds a mirrored copy of a training image. Then another one, with a filter on it. Then a third, cropped and covered in emoji. You run a duplicate check. It finds nothing. That's because duplicate checks look for the same bytes, and an edited copy has different bytes. A mirrored JPEG is a different file from the original. To a hash of the file contents, the two are strangers. To a person, they're the same picture. The task is called image copy detection. Given a query image, decide whether it's an edited copy of an image in a reference collection, and say which one. This post walks through one practical approach: compute two signals per image, combine them with a rule you control, and read the result as a graph where every link comes with evidence. It's packaged as an open-source FiftyOne plugin called Image Link Lab, but the idea carries over to any dataset where copies hide behind edits. !Image Link Lab in FiftyOne. Tune the pHash and CLIP limits, and the copy graph redraws as you drag. Image Link Lab in FiftyOne. Tune the pHash and CLIP limits, and the copy graph redraws as you drag. Key takeaways - Image copy detection asks whether a query image is an edited copy of a reference image, and which one. compute_exact_duplicates can't answer that, because edits change the bytes. - A perceptual hash (pHash) summarizes pixel layout in 64 bits. It tolerates recompression and small blurs and breaks on mirrors, rotations, and big crops. - A CLIP embedding (clip-vit-base32-torch from the FiftyOne model zoo) summarizes what the picture shows. It survives mirrors and crops and gets fooled by look-alikes. - Each signal's blind spot is the other's strength, so the plugin computes both and lets you choose how they combine: link a pair if either signal passes (combine: "any"), or only if both do ("all"). - Moving a limit changes the answer. The Image Link Lab panel redraws the links as you drag the sliders and shows the numbers behind every line. Given a truth field, it scores each rule with FiftyOne's native evaluation so you can compare rules in Model Evaluation. - The output is ordinary FiftyOne data: a phash field, a copy_of Classification field, and copies_* evaluation runs. You can tag copies, save views, and filter a test split cleanly with the code you already use. What is image copy detection? Image copy detection is the task of deciding whether a query image is a modified copy of any image in a reference corpus. The edits can be mundane (crop, resize, or re-compress) or hostile (mirror, overlay text, blend with another image, or screenshot it on a different device). The standard benchmark is DISC21, the dataset behind the Image Similarity Challenge at NeurIPS 2021. Its authors describe the goal as determining "whether a query image is a modified copy of any image in a reference corpus of size 1 million," with edits that include automated transformations, hand-crafted changes, and machine-learning-based manipulations. Both the query set and the reference set are mostly distractors that match nothing, a needle-in-a-haystack setting like real life. The paper is The 2021 Image Similarity Dataset and Challenge. Copy detection is a different question from "which images look like this one?" A similarity search returns neighbors. Copy detection has to commit: this query is a copy of that original, or it isn't a copy of anything. !One beach photo and five edited copies (mirrored, cropped, blurred, text overlay, and grayscale with rotation) stored on five different devices. Exact-duplicate search finds none of them. A side panel lists why finding them matters: train/test leakage, labeling cost, licensing, and forensics. Five edited copies of one image are five different files, so exact-duplicate search finds none of them. Where edited copies cost you You don't need a content moderation team to care about this. Four situations come up again and again: - Train/test leakage. A mirrored copy of a training image in your test set inflates your metrics. You don't find out until the model meets real data. - Labeling cost. You pay to label the same picture several times, sometimes with conflicting answers. - Licensing and attribution. You need to know where an image came from, and a cropped copy hides its source. - Moderation and forensics. A known-bad image comes back filtered, or shows up on a second device. The first two hit any training pipeline. The last two are what DISC21 was built around: its authors say the benchmark mimics social media problems such as misinformation and objectionable content. Why exact duplicates and single embeddings fall short FiftyOne already handles two nearby problems. compute_exact_duplicates finds byte-identical files. compute_near_duplicates, along with the Similarity Search panel, finds neighbors by a single embedding. Neither is built for edited copies. Exact matching fails the moment a pixel changes. A single embedding gives you a ranked list of look-alikes, with one similarity score per pair and no way to say which kind of evidence you trust. If the score is wrong, you can't tell whether the embedding was fooled or the copy was over-edited. What's missing is a way to combine two different kinds of similarity with a rule you control, and a way to see which image is linked to which, and why. Two signals, two blind spots The approach starts from a simple observation. Pixels and meaning fail on different edits. A perceptual hash (pHash) summarizes an image's pixel layout in 64 bits. Two images whose hashes differ in only a few bits look the same at a glance. It's fast, needs no model, and shrugs off recompression and a small blur. It's defeated by anything that moves pixels around: a mirror, a rotation, or a big crop. A CLIP embedding summarizes what the picture shows. Two images with a high cosine similarity are about the same thing. It survives mirrors and crops. It's fooled by look-alikes: two different photos of the same kind of scene can score higher than a heavily blurred copy of the same photo. !Four DISC21 image pairs scored by pHash and CLIP: a mirrored copy only CLIP catches, a striped and blurred copy only pHash catches, a look-alike CLIP nearly links by mistake, and a flipped, blurred, and filtered copy both signals miss. Pixels and meaning fail on different edits, so pHash and CLIP each catch copies the other misses. The four pairs above show it. A mirrored copy is invisible to pHash and obvious to CLIP. A striped and blurred copy is the reverse. A look-alike that isn't a copy nearly gets linked by CLIP. And a flipped, blurred, filtered copy beats both. Good to know. Why not train a copy-detection model? You can, and the DISC21 challenge exists to push that line of work. The two-signal approach is for the common case where you have a dataset on disk, no labeled copy pairs, and a question to answer this afternoon. Both signals work off the shelf. Combine them with a rule you control Each signal gets a limit. For pHash, it's the most bits two hashes may differ by. For CLIP, it's the least cosine similarity two images must reach. Then you choose how the signals combine: - Either signal is enough (any): link the pair if pHash or CLIP passes. You get more links, and more wrong ones. - Both must agree (all): link only if both pass. You get fewer links, and almost all of them right. Moving a limit is the whole game. Loosen the pHash limit, and you catch more heavily edited copies and start linking images that only look alike. Tighten it to all, and the wrong links disappear along with most of the right ones. No setting gets everything. That's why the plugin doesn't give you a single "is copy" score. It shows you where the trade-off sits for your data and lets you pick. Keep it honest. There is no universal threshold. A limit that works on re-uploaded product photos will behave differently on a collection full of heavy crops and overlays. If you have even a small set of known copy pairs, use it to calibrate the rule. Where FiftyOne and its plugin ecosystem fit FiftyOne is an open-source tool for building high-quality datasets and visual AI models. It gives you the dataset layer: samples with fields you can filter and sort on, views, tags, the App for looking at your images, and native tools for similarity search and model evaluation. It can't ship every check that depends on your data. That's the job of plugins. A FiftyOne plugin is a folder FiftyOne discovers at runtime. It includes a manifest (fiftyone.yml), Python code, and optionally a React front end. You add a button, a form, a background job, or a whole panel to the App without forking FiftyOne or waiting for a release. Plugins come in two kinds: - Operators are actions. You trigger one, it runs, it finishes. - Panels are UI that stays open next to your data and reacts when you filter the grid. Plugin code runs on the FiftyOne server in Python so that it can use NumPy, PyTorch, a GPU, or any library you can pip install. Long jobs can be delegated to a background worker so the App stays free. Every operator also has a URI, so anything that can call it, from a button to a notebook to an AI agent, runs the same code. For a deeper tour of how operators and panels work, see Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins. Image Link Lab is built from standard parts: two operators (compute_signals, find_copies), one hybrid panel (image_link_lab), native similarity indexes, and native evaluation runs. The plugin adds only what FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph. !The five-step Image Link Lab workflow: compute signals, find copies, open the panel, score the rule, and act. Each step lists what is plugin code and what is native FiftyOne. The plugin adds three things FiftyOne lacks: a perceptual hash, a rule over two signals, and the graph. Everything else is native. Step 1: Install the plugin and compute signals Install the plugin and its requirements: fiftyone plugins download https://github.com/harpreetsahota204/image_link_lab fiftyone plugins requirements @harpreetsahota/image_link_lab --install The plugin needs fiftyone>=1.22.1. CLIP runs on CPU but is much faster on a GPU. Open any image dataset in the App, click + next to Samples, and add the Image Link Lab panel under Custom. Click Compute signals, keep both boxes ticked, and execute. !Computing the pHash and CLIP signals with the Image Link Lab operator. Computing the pHash and CLIP signals with the Image Link Lab operator. The operator writes three things: | Where | What | | --- | --- | | phash field | the 64-bit pHash of each image, as hex | | brain key phash | the hash bits as a native similarity index, so sort-by-similarity in the App works on pHash too | | brain key clip | a native CLIP similarity index (clip-vit-base32-torch from the model zoo) | What the compute signals operator writes to your dataset: a phash field and two native similarity indexes. It runs delegated by default, so start a worker with fiftyone delegated launch or choose "Execute now" in the form. pHash takes about a minute per 10,000 images, plus CLIP time. Signals that are already computed are left alone unless you tick them. Step 2: Find copies Click Find copies and fill in the form. !Filling in the find copies form: queries, candidate originals, an optional truth field, and the starting rule. Filling in the find copies form: queries, candidate originals, an optional truth field, and the starting rule. - Query images are the ones you want explained. Each is linked to at most one original. - Candidate originals are where to look. Both can be the current view, the whole dataset, or a tag. They can overlap, and an image is never linked to itself. - ID field (optional) is what to call each image, if your dataset identifies images by something other than the sample ID. - Truth field (optional) holds, for each query, the ID of its known original. This is your ground truth, supplied by your dataset, not computed by the plugin. Set it, and you get scoring. Leave it empty, and the plugin works unsupervised: links are drawn, nothing is judged. - Rule sets the starting pHash limit, CLIP limit, and any or all. You'll change these in the panel, so this is just where the first result comes from. Behind the form, the operator does three things: 1. Gathers candidates. For each query, it takes the 5 nearest originals by pHash and the 5 nearest by CLIP, plus the true original if a truth field is set. It stores both signals' values for every candidate. This is the slow step, about 4 seconds for 1,000 queries against 5,000 originals, and every later run reuses it. 1. Applies the rule and writes copy_of, a Classification per query whose label is the linked original's ID, or none. copy_of is a prediction from your rule, not a human label. When several candidates pass, the one passing the most signals wins, then the highest CLIP similarity, then the fewest differing bits. Scores, if a truth field is set. !The find copies form beside the four steps its execute function performs: gather candidates once, apply the rule, score it with a native binary evaluation, and remember the settings for the panel. The slow step, gathering candidates, runs once. Every rule you try after that reuses them. Step 3: Read the graph The panel draws whatever queries are in the current grid view, up to 40 at a time. Filter the grid and the graph redraws. - Nodes are images. Copies have a gray border. Originals have an orange border and, when more than one copy points at them, a badge with the count. - Lines are candidate pairs the rule has something to say about. With a truth field set, green is a correct link, red is an incorrect link, and dashed amber is a missed original. Thick lines pass both signals, and thin lines pass one. - Images joined by lines form families, laid out as a star with the original in the middle. A loose rule joins families through wrong links, and you'll see red lines bridging two stars. - Queries with nothing to draw sit in a strip at the bottom. For a distractor with no original, that's the correct outcome. Click an image and the grid filters to that image and everything it has a line to, including originals your current view doesn't contain. Click a line and an evidence pane opens with the copy and the original side by side, then one row per signal: pHash pixels 30 bits differ limit ≤ 10 fails CLIP meaning 0.731 similar limit ≥ 0.90 fails Below the rows is a verdict in a sentence, such as "Not linked: pHash and CLIP are outside the limit. This is the true original, so the rule missed it." You can check every statement against the numbers above it. No model decides behind the scenes. Move the sliders. The rule runs in the browser against the stored candidate values, so lines and counts update as you drag, with no server round trip. Good to know. Does the graph work without ground truth? Yes. Without a truth field, every link is drawn green, and nothing is dashed. The colors mean "linked," not "right." Step 4: Score the rule and compare alternatives If you set a truth field, click Score this rule. The plugin runs the rule over every query, not just the 40 on screen, writes copy_of, and records a native FiftyOne evaluation named after the rule, such as copies_phash10_or_clip90. Then it opens Model Evaluation. The evaluation is binary, per query. Ground truth is copy if the query has a known original and unique if it doesn't. The prediction is copy if the rule linked the query to its true original, and unique otherwise. FiftyOne computes precision, recall, F1, and a 2×2 confusion matrix from those two labels. One definition matters: a copy linked to the wrong original counts as a miss, not a false positive. The evaluation asks whether you found the true original, and the answer is no. The loop looks like this: 1. Score the starting rule and open its evaluation in Model Evaluation. 1. Click the false-negative cell. The grid filters to the misses, and the graph redraws them with dashed amber lines to their originals. Click a line to see which signal failed and by how much. That tells you which slider might help. 1. Change a limit, score again, and you get a second evaluation key. 1. Open one and compare it with the other. 1. Mark the rule you keep as Reviewed and delete the rest with dataset.delete_evaluation(key). The false-positive cell is the other one worth clicking. Those are distractors the rule linked, usually look-alikes, which is CLIP's failure mode. Comparing a pHash-only run with a CLIP-only run shows which edits each signal survives. Keep it honest. Model Evaluation scenarios, which break results down by edit type, are a Voxel51 feature. In open-source FiftyOne, filter the grid by an edit-type field and read the chips, or compute the numbers in Python. Step 5: Act on the results copy_of is an ordinary label field, so everything else in FiftyOne works on it: from fiftyone import ViewField as F copies = dataset.match(F("copy_of.label") != "none") copies.tag_samples("copy") dataset.save_view("copies", copies) # Keep a test split clean clean_test = dataset.match_tags("test").match(F("copy_of.label") == "none") # Everything linked to one original dataset.match(F("copy_of.label") == "R003513") Typical setups differ only in what you pass as queries and originals: | Situation | Queries | Originals | Truth | | --- | --- | --- | --- | | Dedup a collection | whole dataset | whole dataset | none | | Check a test split for leakage | tag test | tag train | none | | Match seized images against a known-bad list | the seized images | the known-bad list | none, or a case label | | Benchmark a rule | the edited images | the originals | the original's ID | Typical Image Link Lab setups, with what to pass as queries, originals, and truth for each. Run the same workflow from Python Everything the App does is an operator, so a notebook, a script, or an agent can run it: import fiftyone as fo import fiftyone.operators as foo dataset = fo.load_dataset("disc21_link_lab") # Step 1: signals (delegate=False runs it in this process) foo.execute_operator( "@harpreetsahota/image_link_lab/compute_signals", {"dataset": dataset, "params": {"phash": True, "clip": True, "delegate": False}}, ) # Step 2: copies, with a rule, scored against a truth field result = foo.execute_operator( "@harpreetsahota/image_link_lab/find_copies", { "dataset": dataset, "params": { "queries": "tag:final_query", # or "view", "dataset", "tag:<tag>" "originals": "tag:reference", "id_field": "disc21_id", # optional "truth_field": "reference_id", # optional; turns on scoring "phash_max": 10, "clip_min": 0.90, "combine": "any", # "any" or "all" "delegate": False, }, }, ) print(result.result) # The results are ordinary FiftyOne fields and runs dataset.load_evaluation_results("copies_phash10_or_clip90").print_report() Run find_copies again with a different rule and the candidates are reused, so it takes about a second and adds another evaluation run to compare. !One operator URI per step, callable from panel buttons, the find copies form, a notebook, an AI agent, or a background worker. Native tools close the loop between the grid, the Image Link Lab panel, scoring, and Model Evaluation. Every step is an operator with one URI, so the App, a notebook, and an AI agent all run the same code. Both operators have typed inputs and show up through the FiftyOne Model Context Protocol (MCP) server (list_operators, get_operator_schema, execute_operator), so an AI agent can compute signals, try a rule, and read back precision and recall. How it's built The plugin is small on purpose. The Python operators and the pure-Python core (hashing, candidate gathering, the rule) are tested with pytest. The panel UI is built in React with VOODO, the component library the FiftyOne App itself uses. The rule exists twice on purpose: in Python for the operators and in TypeScript so the sliders can run it in the browser. A shared fixture, tests/fixtures/rule_cases.json, feeds both test suites, so the two can't drift. !The image_link_lab plugin folder and its four layers (browser, framework, engine, and core), each calling only the layer below it, with the rule mirrored between rule.py and rule.ts. Four layers, each calling only the one below. A shared test fixture keeps the Python and TypeScript rules in sync. Everything else is native FiftyOne. CLIP comes from compute_similarity, the pHash bits are registered as a second similarity index, results are plain Classification fields, scoring is evaluate_classifications, and rule history lives in evaluation runs. Keep it honest. The demo dataset, disc21_link_lab, is a subset of DISC21: 1,000 edited queries and 5,000 originals, with a known original for 500 of the queries. The rest are distractors. The full benchmark has a reference corpus of 1 million images, so don't read results from this subset as benchmark numbers. DISC21's edits are also deliberately brutal (heavy crops, overlays, and blends), so recall will be low on it. On a collection of ordinary re-uploads, pHash alone will catch most copies. Try it yourself - Image Link Lab plugin on GitHub - Try the plugin on the DISC21 dataset on this Hugging Face space - FiftyOne on GitHub - FiftyOne plugin docs - The 2021 Image Similarity Dataset and Challenge (paper) - Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins Frequently asked questions What is image copy detection? Image copy detection is the task of deciding whether a query image is an edited copy of an image in a reference collection, and identifying which one. It differs from similarity search, which returns neighbors without committing to a match. How do I find edited or modified copies of an image in a dataset? Compute more than one kind of similarity, since no single signal survives every edit. Image Link Lab computes a perceptual hash in the phash field and a CLIP similarity index, then links pairs with a rule over both. find_copies writes the result to a copy_of field. Can perceptual hashing detect mirrored or cropped images? Not reliably. A pHash summarizes how pixels are laid out, so anything that moves pixels around (a mirror, a rotation, or a big crop) defeats it. A CLIP embedding survives those edits, which is why the plugin uses both. What's the difference between exact duplicates, near-duplicates, and copy detection in FiftyOne? compute_exact_duplicates finds byte-identical files. compute_near_duplicates finds neighbors by a single embedding. Image Link Lab links each query to at most one original using two signals and a rule you set, and shows the evidence for every link. How do I check a test split for train/test leakage? Run find_copies with queries set to tag:test and originals set to tag:train. Then match copy_of.label != "none" on the test split to see which test images link to a training image. Do I need ground truth labels? No. Without a truth_field, the plugin draws links, and nothing is judged. With one, you get right, wrong, and missed colors, counts, and a native evaluation per rule. How large a dataset can it handle? Candidates are gathered with dense distance matrices, which works fine for tens of thousands of originals. Beyond that, narrow the originals to a tag or a view. Can I use a different embedding than CLIP? Not from the form. The plugin is deliberately opinionated: one signal for pixels, one for meaning. To swap models, change CLIP_MODEL in constants.py to any zoo model that produces embeddings. Is CLIP slow? It's the only model in the pipeline, and it runs once. On CPU, budget about a second per image. On a GPU, a few minutes for several thousand.

50 Embodiments, 497 Episodes, One Indexed Dataset: Curating the LeRobot Community Dataset with FiftyOne A single prompt to an AI agent running FiftyOne Skills pulled 10 episodes from every robot embodiment in the LeRobot community dataset, computed video embeddings with Qwen3-VL, and built a queryable similarity index. Here's the code and the thinking behind it. The LeRobot community dataset is one of the largest community-built pools of cross-embodiment robot data: 235 contributors, 50 robot types, ~900 GB. Hugging Face's dataset card lists 46 robot types across 791 cleaned datasets. The 50 here come from scanning robot_type in all 1,755 contributor folders, so the two counts cover different sets of datasets. It's organized as 1,755 contributor datasets, one per folder, so picking a balanced slice starts with reading each one's metadata. If you're training a policy that has to generalize across robots, you need episodes from every embodiment and a way to see which clips are actually worth keeping. Now that FiftyOne natively supports LeRobot, I figured I’d use the FiftyOne skill to import LeRobot datasets and check it out. All it took was a one-sentence prompt: "i want to have a little bit from every embodiment, i have enough disk space for the whole dataset... but let's take 10 episodes from each embodiment to start" That sentence kicked off a session that read every contributor dataset's metadata, found 50 unique robot types, downloaded the right shard files for each, imported 497 episodes into FiftyOne, embedded every clip with Qwen3-VL, computed a similarity index, and saved 75 queryable views. All without writing a data loader, a Parquet parser, or a video decode loop. One sentence was enough because the agent had FiftyOne Skills installed. Skills are packaged workflows that teach AI assistants how to do complex FiftyOne tasks: importing LeRobot episodes, running zoo models, computing embeddings, finding duplicates. The agent didn't have to guess at the LeRobot v3 layout or the FiftyOne API. It followed the skill. Install them in Cursor, Claude Code, Codex, or Gemini CLI with one command: curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills This is what FiftyOne is for. Not visualization. Dataset curation at the scale that robot learning actually requires. New here? FiftyOne is an open-source toolkit for loading, querying, visualizing, and curating ML datasets — images, videos, and multimodal data. LeRobot is Hugging Face's framework for robot learning: it defines a standard v3 dataset format (Parquet + MP4 shards with episode metadata) that the community uses to share teleoperation recordings. Key takeaways - lerobot/community_dataset_v3 is a collection of 1,755 contributor datasets across 50 robot embodiments, each a complete LeRobot v3 dataset with its own meta/info.json. - Using the FiftyOne Skills dataset-import workflow, the agent scanned every */*/meta/info.json, grouped by robot_type, picked the smallest source dataset per embodiment with ≥10 episodes, and downloaded only the needed shard-0 files. - Each FiftyOne sample is one episode — not a frame, not a video file — with fields robot_type, task, duration, fps, dataset_name, and a media_reference pointer into the local LeRobot directory tree. - Qwen3-VL-Embedding-2B turns each episode (up to 32 frames from its time window) into one 2048-dimensional video embedding. The similarity index, Uniform Manifold Approximation and Projection (UMAP), uniqueness, and representativeness all run from that one field in under a minute. - The same model embeds text into the same space, so the index answers text queries: type "folding a cloth" and get the same bimanual cloth-folding episodes back, with no labels involved. - uniqueness scores range from 0.013 to 1.0 across the 497 episodes. 292 clips score below 0.1 because each embodiment's 10 clips come from one recording session. The two highest scores, the only ones above 0.7, both belong to single-frame episodes (0.02–0.03 s long): uniqueness flags broken recordings as readily as rare behavior. - 75 saved views — 50 by robot type plus analytical cuts by frames per second (FPS), duration, robot category, task keyword, and curation intent — are stored in the dataset and load from the App's Views dropdown with no code. What you need To load and query the pre-built dataset (no GPU required): pip install "fiftyone>=1.22.0" "huggingface_hub[cli]" transformers qwen-vl-utils torch torchvision To re-run UMAP visualization yourself (Step 3): pip install umap-learn FiftyOne 1.22.0 is the first release with fo.types.LeRobotDataset. The dataset loads from the Hub with embeddings, similarity index, UMAP layout, and curation scores already computed. Steps 1–2, Step 4, and episode-to-episode search in Step 5 run entirely on the pre-built data, with no GPU or model weights. Text search in Step 5 downloads Qwen3-VL-Embedding-2B (about 4 GB) to embed your query; it runs on CPU and is faster on a GPU. Step 3 is for readers building their own dataset who want to replicate the embedding pipeline. How the LeRobot community dataset is organized !FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values lerobot/community_dataset_v3 is a collection of contributor datasets. Each one is a complete LeRobot v3 dataset in its own <contributor>/<dataset>/ folder: community_dataset_v3/ kuehnrobin/ g1_stack_cube_left/ meta/info.json data/ videos/ villekuosmanen/ pick_coffee_capsule_under_dome/ meta/info.json ... 1,755 directories. 17,617 files. 897.5 GB total. Each sub-dataset carries its own meta/info.json with its robot type, episode count, and FPS. fo.types.LeRobotDataset reads one LeRobot v3 dataset at a time, so the agent started by downloading all */*/meta/info.json files — a 56-second fetch — and then scanning them to build the plan: # Counts across all 1,755 sub-datasets Robot types (50 unique): so100 899 datasets 34801 total_episodes so101 312 datasets 7490 total_episodes so101_follower 121 datasets 4414 total_episodes so100_follower 120 datasets 3065 total_episodes arx5 43 datasets 2108 total_episodes Unitree_G1_inspire 1 dataset 7 total_episodes # only 7 available so100MovellaDot 75 datasets 157 total_episodes # max 5 per source ... All 1,755 sub-datasets were already at codebase_version: v3.0. No conversion needed. !Diagram of the contributor and dataset folder layout, the four-step plan to scan, group by robot type, pick one source per embodiment, and download, and the shard-0 Parquet and MP4 files fetched from one source Left: the contributor/dataset folder layout, with the per-dataset meta/info.json files the agent scans first. Middle: the four-step plan — scan, group by robot_type, pick one source dataset per embodiment, download. Right: inside one source dataset, only file-000.parquet and file-000.mp4 (the first ~10 episodes) are fetched; the other shards are skipped. Good to know. What's a shard-0 set? LeRobot v3 packs many episodes into shared MP4 and Parquet shards. file-000.parquet and file-000.mp4 hold the first batch of episodes for each dataset. Downloading only those files makes a 10-episode subset cheap, even from a 50-GB source dataset. Step 1: load the dataset The agent session did the heavy lifting — reading every contributor dataset's metadata, picking the right shard files, importing 497 episodes, computing embeddings, and pushing the result to the Hub. The import step is covered in How to Import LeRobot Datasets into FiftyOne Using Agent Skills. The dataset is at Voxel51/community_v3_10per_embodiment. Load it: import fiftyone as fo import fiftyone.utils.huggingface as fouh ds = fouh.load_from_hub("Voxel51/community_v3_10per_embodiment") print(len(ds)) # 497 print(ds.distinct("robot_type")) # 50 embodiments print(ds.get_field_schema()) # episode_index, robot_type, task, duration, fps, # dataset_name, media_reference, # qwen3vl_embedding, qwen3vl_embedding_camera, # uniqueness, representativeness, ... From here on, the source videos live in the Hub repo alongside the metadata. FiftyOne reads through the media_reference pointer — nothing to download separately. And because the unit is an episode rather than a file, every query you run is a dataset-level question, not a file-browser search. What "one sample = one episode" actually means !A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference Most tools treat robot data as video files or frame sequences. FiftyOne takes a different unit: the episode. Every sample in this dataset carries: sample.episode_index # int — 0..9 within the source dataset sample.robot_type # str — e.g. "arx5", "so100", "Unitree_G1_Dex3" sample.dataset_name # str — e.g. "villekuosmanen/pick_coffee_capsule_under_dome" sample.task # str — e.g. "Pick up the coffee capsule" sample.duration # float — seconds (0.02 to 148.6 in this dataset) sample.fps # float — 10, 15, 20, 25, 30, or 50 sample.media_reference # LeRobotEpisodeReference — a FiftyOne type with two fields: # .data = [chunk_index, file_index, first_row, last_row] (Parquet) # .videos = {camera: [chunk_index, file_index, from_ts, to_ts]} (MP4) Think of a FiftyOne sample as a row in a table: the left side has filterable scalar fields (robot_type, task, duration, fps) and the right side has the media_reference pointer. The pointer's two values — data[2:4] and videos[top][2:4] — map to the actual Parquet rows and MP4 time window in the LeRobot folder on disk. !Diagram of a FiftyOne sample with filterable scalar fields above a media_reference pointer whose data and videos values map to Parquet rows and an MP4 time window on disk A FiftyOne sample (left) with filterable scalars above the line and the media_reference pointer below it. Arrows connect the pointer's two values — data[2:4] and videos[top][2:4] — to the actual Parquet rows and MP4 time window in the LeRobot folder on disk. Three roads lead from a FiftyOne view back to frame data: !Diagram of three ways to get frames from a FiftyOne view: read in place and Torch dataset, which copy no data, and export to a standalone LeRobot v3 dataset, which copies data 1. Read in place via sample.media_reference.data — no copy, your code, one episode at a time. 1. Torch dataset via view.values("episode_index") passed to LeRobotDataset(..., episodes=...) — no copy, for training loops. episode_index restarts at 0 in every source dataset, so filter the view to one dataset_name first; for a mix of sources, export instead 1. Export via view.export(dataset_type=fo.types.LeRobotDataset) — copies data, produces a standalone v3 repo. Each export must come from a single source dataset, so filter by dataset_name first. The rule: in-process → don't copy. Out-of-process → copy. | Path | Call | Copies data? | Use it when | | --- | --- | --- | --- | | Read in place | sample.media_reference.data | No | Your own code needs one episode's rows, such as for a plot, a metric, or a policy | | Torch dataset | LeRobotDataset(..., episodes=view.values("episode_index")) | No | A training loop reads batches from a curated subset filtered to one dataset_name | | Export | view.export(dataset_type=fo.types.LeRobotDataset) | Yes | Something outside the Python process needs a standalone LeRobot v3 repo, one source per export | Three ways to get frames from a FiftyOne view of LeRobot episodes, and when each one copies data From inspection to curation !Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid Timeline viewers are built for inspecting one recording at a time: open a log, scrub the timeline, and watch camera, inertial measurement unit (IMU), and joint streams in sync. That's the right tool when you're debugging one specific run. FiftyOne starts where that ends. Once a run is recorded and you have hundreds of episodes, the question isn't "what happened in this episode?" It's "across all 497 episodes from 50 different robots, which clips are visually unique, which are near-duplicates, which ones actually belong in my training set?" A timeline scrubber can't answer that. A queryable, indexed dataset can. The rest of this post shows exactly what that looks like — every step from here on is something you can't do in a timeline scrubber. Step 2: what's already on the dataset The Hub dataset ships with a video embedding for every episode, computed with Qwen/Qwen3-VL-Embedding-2B from the qwen3vl_embeddings remote zoo source. Each episode's time window is decoded with PyAV; up to 32 evenly spaced frames go to the model as one video clip, producing a 2048-dimensional qwen3vl_embedding field. qwen3vl_embedding_camera records which camera stream the frames came from, since camera names and layouts differ across the 50 embodiments. What Qwen3-VL-Embedding actually produces, and why it matters Qwen3-VL is an open vision-language model family from Qwen/Alibaba. It comes in two variants: Qwen3-VL-Instruct (generates text — captions, bounding boxes, answers) and Qwen3-VL-Embedding (trained for retrieval, so it produces dense vectors directly). The embedding model reads the clip, and the hidden state of the last token becomes the vector, L2-normalized: # what produced qwen3vl_embedding (inside the qwen3vl_embeddings zoo model) outputs = model(**inputs) # frames + instruction embedding = outputs.last_hidden_state[row, last_token_position] # one 2048-d vector per clip embedding = F.normalize(embedding, p=2, dim=-1) !Two-lane diagram in which frames decoded with PyAV pass through Qwen3-VL-Embedding into one 2048-dimensional vector, and a text query passes through the same model into the same vector space Top lane: an episode's time window is decoded with PyAV, up to 32 evenly spaced frames are tokenized by the vision encoder and run through the language model, and the last token's hidden state becomes one 2048-d vector stored as qwen3vl_embedding. Bottom lane: at query time, a text query goes through the same model and lands in the same space, which is what makes text-to-video search work. This model is useful for robotics data because video and text map into the same vector space. The similarity index qwen3vl_embedding_sim keeps the model name, so it accepts text queries as well as episodes: ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim") — no labels, no metadata matching, just the model's understanding of what was in the clip. For Vision-Language-Action (VLA) model development, this is specifically useful at the data curation stage. A VLA policy needs to generalize across scenes, embodiments, and task variations. But you can only curate what you can measure. Video embeddings give you a numeric handle on task semantics and visual appearance simultaneously, which makes it possible to compute: - Similarity — which episodes, across all 50 embodiments, are doing visually similar things? - Uniqueness — which demonstrations cover behavior your dataset hasn't seen much of? - Representativeness — which episodes best stand in for entire clusters of similar behavior? You can't extract these signals from episode metadata alone. They require a model that actually watched the clips. You also get uniqueness and representativeness scores out of the box. More on those below. If you want to try a different model, such as Qwen/Qwen3-VL-Embedding-8B, write its vectors to a new field; Step 5 shows the full embedding loop for LeRobot episodes. Step 3: Compute similarity, uniqueness, and representativeness !FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram Skip the code if you loaded from the Hub. The pre-built dataset already includes qwen3vl_embedding_sim, qwen3vl_embedding_umap, uniqueness, and representativeness. The code block below is for readers building their own LeRobot dataset — it requires umap-learn and an existing embedding field. The explanations after it apply either way. If you're building your own dataset, compute the qwen3vl_embedding field first (the loop is in Step 5), then run the four calls: import fiftyone.brain as fob EMBED = "qwen3vl_embedding" # Similarity index — powers Sort by Similarity in the App. Passing the model name # lets the index embed text queries (register the qwen3vl_embeddings source first) fob.compute_similarity( ds, embeddings=EMBED, model="Qwen/Qwen3-VL-Embedding-2B", brain_key="qwen3vl_embedding_sim", ) # UMAP — 2D layout for the Embeddings panel fob.compute_visualization( ds, embeddings=EMBED, method="umap", brain_key="qwen3vl_embedding_umap", num_dims=2, seed=51, ) # Uniqueness — how different each episode is from its nearest neighbors fob.compute_uniqueness( ds, embeddings=EMBED, uniqueness_field="uniqueness", similarity_index="qwen3vl_embedding_sim", ) # Representativeness — how close each episode is to a cluster center fob.compute_representativeness( ds, embeddings=EMBED, representativeness_field="representativeness", similarity_index="qwen3vl_embedding_sim", ) Why UMAP for video embeddings? Each episode is a 2048-dimensional vector. You can't look at that. Dimensionality reduction maps it to 2D so you can see structure in the Embeddings panel. The three common choices are principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and UMAP: - PCA is linear. It finds the axes of maximum variance, which is fast but can't capture the curved manifold structure that emerges when you embed semantically diverse video. A pick-and-place cluster and a folding cluster may sit on a low-variance direction PCA ignores entirely. - t-SNE is non-linear and good at revealing local clusters, but it destroys global structure — the distance between clusters in the 2D plot is meaningless, and it's slow on datasets larger than a few thousand samples. - UMAP is non-linear, keeps local cluster structure, and usually keeps more of the arrangement between clusters than t-SNE does. Don't over-read it, though: distances between far-apart clusters are still only roughly meaningful. It runs on 500 episodes in seconds. FiftyOne's UMAP backend uses Euclidean distance with min_dist=0.1 by default, which gives tight clusters. !Three scatter plots of the same 497 Qwen3-VL-Embedding vectors reduced with PCA, t-SNE, and UMAP, colored by robot category The same 497 Qwen3-VL-Embedding vectors reduced three ways, colored by robot category. PCA's two axes hold 20% of the variance, and the categories overlap; t-SNE separates local knots but inter-knot distances are arbitrary; UMAP is the layout the Embeddings panel shows. In every map, the 10 clips from one source dataset collapse into a single knot—same scene, same camera, same task. For a first look at a cross-embodiment dataset, UMAP is the practical choice: you get clusters you can click into, and their rough arrangement is meaningful. The map also shows what the embeddings respond to most: the recording session. For 495 of the 497 episodes, the nearest neighbor comes from the same source dataset. Step past a clip's own siblings, though, and the next matches in the full 2048-d space often share the task on a different robot. The nearest non-sibling to the so100 box-in-bin episodes is a so100MovellaDot ball-in-box episode (cosine distance 0.157), and the nearest non-sibling to the arx5 dome-lifting episodes is a trossen_ai_stationary cube-in-box episode (0.170). What uniqueness and representativeness actually compute Both metrics are grounded in the embedding space. The FiftyOne Brain docs cover uniqueness and representativeness; here's what each one computes: Uniqueness — K-nearest-neighbor distance, K=3. For each episode, FiftyOne finds its 3 nearest neighbors in embedding space and computes a weighted mean of those distances: uniqueness = weighted_mean(distances_to_3_nearest_neighbors) weights = [0.6, 0.3, 0.1] # closer neighbors count more normalized to [0, 1] by dividing by the dataset maximum Interpretation: uniqueness=1.0 means this episode is farther from its nearest neighbors than any other episode in the set — it occupies a region of embedding space that no other clip is near. uniqueness=0.013 means the episode is almost on top of another one — effectively a near-duplicate. !Toy embedding space where an isolated episode scores 1.0 and a clustered one scores about 0.01 on uniqueness, beside a log-scale histogram of uniqueness across 497 LeRobot episodes Left: a toy embedding space. Episode A is far from its three nearest neighbors (line width = weight), so it scores 1.0; episode B sits in a knot of near-identical clips and scores ≈ 0.01. Right: the real distribution over all 497 episodes on a log scale — 292 score below 0.1, two score above 0.7, and one arx5 clip (a single-frame episode) sets the maximum at 1.0. Representativeness — K-Means distance to cluster center, K=20 clusters. For each episode, FiftyOne finds the nearest cluster center and scores it as: representativeness = 1 / (1 + distance_to_nearest_cluster_center) normalized locally within each cluster Interpretation: representativeness=1.0 means this episode is at the heart of its cluster — it's the most "typical" example of that group of behaviors. Low representativeness means the episode sits at the edge of a cluster, more idiosyncratic than central. !K-means cluster centers with equal-distance rings, where a central episode scores near 1.0 and a peripheral one near 0.3, beside the representativeness distribution across 497 episodes Left: K-Means centers (×) with rings of equal distance. Episode P sits on its center and scores ≈ 1.0; episode Q is assigned to the blue cluster but sits far out and scores ≈ 0.3. Right: the real distribution — 198 of 497 episodes score above 0.8, and the floor of 0.33 means no episode is a pure outlier. Why these two metrics are the right pair for VLA data curation A Vision-Language-Action model needs two things from its training data: 1. Coverage — enough behavioral diversity for the policy to generalize to new scenes and task variations. Low-uniqueness episodes (near-duplicates) don't add coverage; they just add training compute and risk overfitting to that one scene configuration. 1. Signal density — enough repetition of each important behavior for the policy to learn it. Keeping only high-uniqueness episodes would discard the repetitions that make a skill learnable. Uniqueness and representativeness together let you navigate that tradeoff explicitly: | Goal | Query | | --- | --- | | Seed set: diverse AND learnable | High uniqueness + high representativeness | | Redundancy audit: what can be cut? | Low uniqueness floor (near-duplicates) | | Canonical demos per task | Sort by representativeness within each task view | | Rare demonstrations (or broken ones) | Uniqueness > 0.7, then check duration | Four VLA data curation goals and the uniqueness and representativeness queries that answer each one. !Scatter plot of uniqueness against representativeness for every LeRobot episode, colored by robot category, with shaded query regions and 50 ringed points marking curate_diverse_seeds Every episode is placed by its two scores (uniqueness on a symlog axis, colored by robot category). The shaded regions are the queries from the table: the redundancy audit on the left, canonical demos along the top, high-uniqueness outliers on the right, and the 50 orange-ringed points are curate_diverse_seeds — the top-50 by uniqueness with representativeness above 0.5. In this dataset, uniqueness ranges from 0.013 to 1.0. The floor shows near-identical clips in the set — 292 of 497 score below 0.1, and the bottom six are four moss cube-stacking episodes and two sam_bimanual cloth-folding episodes, each from a single recording session. At the top, only two episodes score above 0.7: an arx5 clip at 1.0 and an xarm6 clip at 0.86. Both are single-frame episodes, 0.02–0.03 seconds long. representativeness ranges from 0.33 → 1.0 — no episode is a pure outlier; even the most peripheral clips are loosely connected to a cluster. Keep it honest. Both ends of the uniqueness ranking come from how we built this subset, not from rare behavior or wasted data. Each embodiment contributes 10 consecutive episodes from one recording session, so every clip has nine siblings sitting right next to it in embedding space. On this dataset, every one of the 50 episodes in curate_redundant_candidates has a sibling as its nearest neighbor, sampled on purpose. The top of the ranking has the opposite story: the five single-frame episodes look unlike everything else because there's almost nothing in them. They take the top two spots, and all five land in curate_diverse_seeds. Uniqueness did its job by surfacing recordings worth a second look. Those clips belong in the cut pile, though, not the seed set, so pair uniqueness with a duration filter. These two numbers let you make dataset decisions — what to keep, what to cut, what's overrepresented — without watching a single video. A log viewer shows you one episode at a time. FiftyOne gives you a signal across all 497 at once. Step 4: Save views that answer real questions !Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App Views in FiftyOne are lazy queries — no data is copied, they resolve on load. Once saved, they appear in the App's Views dropdown. The pre-built dataset already has 75 views saved. Check what's there: import fiftyone as fo print(ds.list_saved_views()) # ['so100', 'so101', ..., 'curate_diverse_seeds', 'curate_redundant_candidates', # 'category_humanoid', 'task_pick_place', 'fps_30', ...] To build them yourself on your own dataset: import fiftyone as fo # Curation: 50 most visually distinct AND cluster-representative episodes ds.save_view( "curate_diverse_seeds", ds.sort_by("uniqueness", reverse=True) .match(fo.ViewField("representativeness") > 0.5) .limit(50), ) # Redundancy candidates: bottom 50 by uniqueness ds.save_view( "curate_redundant_candidates", ds.sort_by("uniqueness").limit(50), ) # Robot category views HUMANOID = ["Unitree_G1_Dex3", "Unitree_G1_Gripper", "Unitree_G1_Inspire", "Unitree_G1_inspire", "g1"] ds.save_view("category_humanoid", ds.match(fo.ViewField("robot_type").is_in(HUMANOID))) # Task keyword views ds.save_view("task_pick_place", ds.match(fo.ViewField("task").re_match("pick|place|grasp|grab", options="i"))) # FPS views — reveal recording-condition variation across embodiments for fps_val in [10.0, 15.0, 20.0, 25.0, 30.0, 50.0]: view = ds.match(fo.ViewField("fps") == fps_val) if len(view) > 0: ds.save_view(f"fps_{int(fps_val)}", view) 75 views total. fps_30 covers 387 of the 497 episodes. fps_10 covers 30 — all from three stationary arms: widowx, ur5e_gello, and mycobot320. task_pick_place returns 367 episodes; task_fold_pour returns 70. Each is a reusable, shareable slice of the dataset that resolves in milliseconds. No script, no re-download, no custom data loader — just a name in a dropdown. Step 5: similarity search by episode and by text !Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel Search by episode The qwen3vl_embedding_sim index powers episode-to-episode search directly from the App or from Python. Pass any sample's ID, and FiftyOne returns the nearest neighbors by video embedding distance. Because every clip's closest matches are its own recording session, filter out the query's dataset_name to see what other robots are doing something similar: from fiftyone import ViewField as F # one sam_bimanual cloth-folding episode as the query query = ds.match( (F("dataset_name") == "girardijp/sam_fold_cloth_single") & (F("episode_index") == 0) ).first() # unfiltered: the query and its 9 siblings from the same session same_session = ds.sort_by_similarity(query.id, k=10, brain_key="qwen3vl_embedding_sim") # filtered: the 10 nearest episodes from other source datasets others = ( ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim") .match(F("dataset_name") != query.dataset_name) .limit(10) ) for s in others: print(f"[{s.robot_type}] {s.task}") # [so100_bimanual] playing with folding (all 10 hits) !A sam_bimanual cloth-folding query episode marked on the UMAP, beside the 10 nearest episodes from other source datasets ranked by cosine distance, all so100_bimanual folding clips Left: the query (star) on the UMAP, with its session's knot and the cross-source neighbors. Right: the 10 nearest episodes from other source datasets, ranked by cosine distance. The query's 9 siblings all sit within 0.015; past them, every hit is a so100_bimanual "playing with folding" episode, 0.19–0.31 away. !Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel A sam_bimanual rig folding a cloth and a pair of SO-100 arms folding cloth look nothing alike as robots, and the model still puts them next to each other. That's the cross-embodiment signal. Some sessions pull in a neighboring robot type instead: the 10 Unitree_G1_Dex3 cube-stacking clips sit within 0.030 of each other, and the first other episode, at rank 10, is a Unitree_Z1_Dual red-cup pick (0.142). Search by text !Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes The same index takes text. The dataset stores the episode vectors, but FiftyOne still needs the model to embed your query, so register the qwen3vl_embeddings remote zoo source first: import fiftyone.zoo as foz # declare which models the remote source provides foz.register_zoo_model_source( "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True, ) # optional: the first text query downloads the weights automatically foz.download_zoo_model("Qwen/Qwen3-VL-Embedding-2B") for query in ["folding a cloth", "pouring water into a cup", "humanoid robot picking up a can"]: view = ds.sort_by_similarity(query, k=3, brain_key="qwen3vl_embedding_sim") print(query, "->", [(s.robot_type, s.task) for s in view]) # summarised output: # folding a cloth -> sam_bimanual, "Fold a cloth" (x3) # pouring water into a cup -> bi_so100_follower, "bi soarm100 pour water" (x3) # humanoid robot picking up a can -> Unitree_G1_Inspire, "Pick up the cans into the box on the table." (x3) The same text box works in the App's similarity panel once qwen3vl_embedding_sim is selected. Not every query lands: "robot stacking a cube" returns a Unitree_Z1_Single red-cup pick rather than the Unitree_G1_Dex3 cube-stacking clips. Treat text search as a fast way to find candidates, then check them. Build the embedding and index on your own dataset LeRobot samples have no filepath, so compute_embeddings can't read the clips for you: decode each episode's time window with PyAV and pass the frames to the model as one video. This is the code behind qwen3vl_embedding (set ROOT to the folder that holds your <contributor>/<dataset>/ directories): from pathlib import Path import av import fiftyone.brain as fob import fiftyone.zoo as foz ROOT = Path("/path/to/community_dataset_v3") def episode_frames(video_path, from_ts, to_ts, max_frames=32): frames = [] with av.open(str(video_path)) as container: stream = container.streams.video[0] fps = float(stream.average_rate or 30) step = max(1, int((to_ts - from_ts) * fps) // max_frames) container.seek(int(from_ts * 1_000_000), any_frame=False, backward=True) i = 0 for frame in container.decode(stream): ts = float(frame.pts * stream.time_base) if ts < from_ts: continue if ts > to_ts or len(frames) >= max_frames: break if i % step == 0: frames.append(frame.to_image()) i += 1 return frames foz.register_zoo_model_source( "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True ) model = foz.load_zoo_model("Qwen/Qwen3-VL-Embedding-2B") # or Qwen/Qwen3-VL-Embedding-8B if model._embedder is None: model._load_model() for sample in ds.iter_samples(autosave=True, progress=True): cam, (chunk, file, from_ts, to_ts) = next(iter(sample.media_reference.videos.items())) path = ROOT / sample.dataset_name / "videos" / cam / f"chunk-{chunk:03d}" / f"file-{file:03d}.mp4" frames = episode_frames(path, from_ts, to_ts) # model.embed() expects a file path; the underlying embedder accepts a frame list as a video sample["qwen3vl_embedding"] = model._embedder.process([{"video": frames}])[0].cpu().float().numpy() sample["qwen3vl_embedding_camera"] = cam fob.compute_similarity( ds, embeddings="qwen3vl_embedding", model="Qwen/Qwen3-VL-Embedding-2B", brain_key="qwen3vl_embedding_sim", ) Passing the model name to compute_similarity lets the index embed text at query time; an index built without it accepts only episodes as queries. On a GPU, the 497 episodes take about 20 minutes. Then run the UMAP, uniqueness, and representativeness calls from Step 3. Putting it together: export a curated LeRobot v3 training split Once loaded, querying across 50 embodiments looks like this: # 50 most unique AND representative episodes, minus the single-frame recordings seed_view = ds.load_saved_view("curate_diverse_seeds").match(fo.ViewField("duration") > 1) # Episodes from other robots that look like a given clip query = ds.match(fo.ViewField("robot_type") == "sam_bimanual").first() similar = ( ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim") .match(fo.ViewField("dataset_name") != query.dataset_name) .limit(10) ) # Episodes matching a description (register the qwen3vl_embeddings source first) cloth = ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim") # Export the curated subset as standalone LeRobot v3 repos, one per source dataset for name in seed_view.distinct("dataset_name"): seed_view.match(fo.ViewField("dataset_name") == name).export( export_dir=f"./my_training_subset/{name.replace('/', '__')}", dataset_type=fo.types.LeRobotDataset, ) The export loops over dataset_name because a LeRobot v3 repo has one feature schema, and the 50 embodiments here have different cameras and action dimensions. The 45 seed episodes come from 16 source datasets, so you get 16 repos. That last step is the point. You went from a 900 GB community dataset to a curated, re-indexed LeRobot v3 training split — filtered by visual uniqueness, robot category, task type, or any combination — without writing a parser, a data loader, or a custom video decode loop. A viewer shows you what the robot did. A curated dataset decides what goes into the model. Frequently Asked Questions What does fo.types.LeRobotDataset actually store in FiftyOne's database? Episode-level metadata only: episode_index, task, duration, fps, robot_type, and a media_reference that points into the local LeRobot directory. No pixel data, no Parquet rows. The source files stay where they are; FiftyOne reads through the pointer. Can I use apply_model on a LeRobot dataset in FiftyOne? Not for a standard zoo model — those expect sample.filepath pointing to a video or image file, and LeRobot samples use media_reference instead. The embedding in this post was computed by working around that: PyAV extracts frames from the episode's time window, and the frames go to the model as a list of PIL images. Step 5 shows the full loop. Why does Qwen3-VL-Embedding use last-token pooling? That's how Qwen3-VL-Embedding was trained. It's a decoder-only model with no CLS token, and its retrieval training teaches the final token's hidden state to summarize the whole input, whether it's a video clip or a sentence. Using the same pooling for both is what puts episodes and text queries in one space. Mean pooling over an instruction-tuned model's hidden states gives usable vectors for comparing clips, but not for matching them against text. How do saved views work after load_from_hub? They're stored in the FiftyOne dataset document alongside the samples. ds.list_saved_views() returns all 75 names; ds.load_saved_view("curate_diverse_seeds") returns the 50 episodes matching that query, resolved against whatever embeddings and fields exist locally. How do I export a curated FiftyOne view to LeRobot v3 for policy training? Export a curated view as standalone LeRobot v3 repos, one per source dataset: for name in view.distinct("dataset_name"): view.match(fo.ViewField("dataset_name") == name).export( export_dir=f"./my_curated_subset/{name.replace('/', '__')}", dataset_type=fo.types.LeRobotDataset, ) The exporter refuses to mix episodes from different sources in one repo, since each embodiment has its own cameras and action layout. For each source, it re-indexes episodes from 0, recomputes stats, and writes a self-contained v3 structure that any LeRobot-compatible trainer can read directly.

10 Multimodal MCAP Datasets in 2026: SLAM Benchmarks, Sensor Fusion, and Ground-Truth Object Detection in FiftyOne A field guide to 10 robotics datasets (driving, aquatic, forest, indoor, and synthetic) repackaged as time-synchronized MCAP recordings for FiftyOne’s native multimodal viewer, with the exact sensor suite, label situation, and a direct link to the paper, the dataset, and (where one exists) a live FiftyOne Space for each. A camera on a driving rig fires at 10-20 Hz, its LiDAR spins at 10 Hz, its inertial measurement unit (IMU) ticks at 200-400 Hz, and its GPS/INS updates entirely on its own clock. None of these streams line up into a clean “one image = one label = one sample” unit, and yet that unit is what almost every computer vision tool assumes you have. Robot data doesn’t work that way. A single recording is a bundle of asynchronous, multi-rate streams captured over minutes, not a folder of independent frames. Trying to force it into per-frame samples either throws away the temporal relationship between sensors or multiplies your sample count by however many topics you’re logging. That’s the problem MCAP was built to solve, and it’s why FiftyOne added native multimodal support for it. We pulled together 10 datasets, all repackaged as MCAP episodes on the Voxel51 Hugging Face org, spanning urban driving, handheld SLAM, aquatic navigation, off-road forest robotics, indoor human-robot interaction, and one fully synthetic set. Below: what MCAP actually is, the data curation problems that show up over and over when repackaging real robot logs into it, a full comparison and schema reference, and a domain-by-domain tour of all 10 datasets with links to the paper, the dataset, and the FiftyOne Space where one exists. Key Takeaways - 10 datasets, 89 total episodes, spanning ground-vehicle driving, campus/handheld SLAM, aquatic navigation, off-road forest robotics, indoor social navigation, and one synthetic set (TartanGround). Every one is a FiftyOne multimodal dataset, where each sample is a continuous recording, not a single frame. - Only 3 of 10 carry real, embedded object-level labels: SemanticSpray++ (all 36 episodes fully labeled, 2D camera boxes, 3D LiDAR boxes, LiDAR + radar point-wise semantics), boreas-multimodal (only 6 of its 8 episodes, the boreas-objects-v1 windows, not the 2 driving episodes), and cmht-autonomous-driving (3D + derived 2D detections on all 4 drives). 6 more (yuto-mms-multimodal, oxford-spires-multimodal, canoe-multimodal, fomo-multimodal-sample, treescope-vat0723-multimodal, TartanGround) are raw SLAM/odometry sensor data with zero object annotations. - navwareset sits in between. Pedestrian 3D cuboids exist and are real, but they’re logged as foxglove.SceneUpdate entities inside the MCAP timeline, not a FiftyOne label field, the same modeling choice used for Boreas’s and CMHT’s boxes, since a multi-minute episode has no single fixed-length list a sample-level label field could hold. - Sensor suites go well beyond camera + LiDAR: boreas-multimodal, canoe-multimodal, and fomo-multimodal-sample all carry radar; canoe-multimodal adds imaging sonar; cmht-autonomous-driving adds a thermal/IR camera; fomo-multimodal-sample adds stereo audio. Three datasets (yuto-mms-multimodal, oxford-spires-multimodal, boreas-multimodal) ship survey-grade ground-truth trajectories accurate to 1-4 cm. - Licensing is genuinely mixed: CC BY 4.0 is most common, but oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA (non-commercial), cmht-autonomous-driving is CC0, boreas-multimodal’s license is unlisted/unknown, and yuto-mms-multimodal has a genuine conflict: the project site says CC BY-NC-SA 4.0, the live Zenodo records serve CC BY 4.0. - 4 of 10 have no arXiv preprint at all (yuto-mms-multimodal, navwareset, canoe-multimodal, cmht-autonomous-driving), DOI/journal citation only, so don’t go hunting for a paper ID that doesn’t exist. What is MCAP, and why did it become the physical AI format? MCAP (pronounced “em-cap”) is a modular, serialization-agnostic binary container format for timestamped pub/sub data. A single .mcap file holds any number of named channels (camera frames, LiDAR scans, IMU samples, GPS fixes, transform trees), each tagged with its own schema, all indexed together so a reader can seek to any timestamp without scanning the whole file. Foxglove built it in 2022 after noticing that robotics teams kept burning engineering time reinventing custom binary formats, only to make third-party tooling harder later. It’s now the default log format for ROS 2 (since the Iron release in May 2023), replacing the SQLite3-backed .db3 bags, which struggled with high-throughput sequential writes. The properties that make it a good robotics log format are the same ones that make it a good physical AI training-data format: - Serialization-agnostic. One file can hold ROS 1 messages, ROS 2 messages, Protobuf, JSON Schema, and FlatBuffers side by side. That’s exactly the mix you get on a real rig with a ROS-based LiDAR driver, a vendor SDK camera, and a custom logger. - Self-contained. Message schemas are embedded in the file itself, so an MCAP recorded years ago is still readable without the original robot software stack, a real concern for datasets that are already 1-5 years old. - Append-only writing. Data streams to disk without ever seeking backward, so a recording survives a software crash or power loss mid-capture rather than being corrupted. - Chunk-indexed for fast seeking. A chunk index at the end of the file lets a reader jump directly to a timestamp instead of scanning from the start. That’s the specific mechanic that makes scrubbing a 20-minute episode in FiftyOne’s multimodal viewer responsive instead of painful. - Built-in compression (LZ4 or Zstandard) at the chunk level, without sacrificing indexed random access (mcap.dev; Segments.ai’s comparison; Voxel51’s MCAP glossary entry). Common data curation issues when repackaging real robot logs into MCAP Each of the 10 datasets below started life as raw ROS bags, per-file sensor dumps, or vendor SDK output and had to be converted. The same handful of problems showed up repeatedly across completely unrelated source datasets, worth knowing before you repackage your own robot logs. - No native MCAP schema for polar or non-camera sensors. Radar (boreas-multimodal, canoe-multimodal, fomo-multimodal-sample) and imaging sonar (canoe-multimodal) have no dedicated Foxglove schema. All three decode their radar into a bird’s-eye-view foxglove.RawImage pseudo-image rather than a real camera view, the closest available representation, not a native one. - Broken or placeholder timestamps that require rebasing; not trusting. treescope-vat0723-multimodal’s processed bag stamped every message with the wall-clock time of an offline reprocessing job run weeks after the actual flight, not the flight’s capture time. Left uncorrected, an episode would nominally span that entire gap with real data in only two tiny slivers. cmht-autonomous-driving’s LiDAR and camera topics had a placeholder sec=0 header on every raw message. navwareset’s robot /tf carried real but non-physical roll/pitch/z noise, verified against the same bag’s adaptive Monte Carlo localization (AMCL) pose (which was exactly zero on every message) before flattening it. - Frame-count assumptions that quietly break on real data. cmht-autonomous-driving’s labels are published against a separately numbered “extracted frame” release, and an initial assumption of “labeled frame N = the Nth bag message” held for exactly 1 of 4 drives. The other 3 had extra bag messages that the extraction dropped, growing throughout the drive. The fix required content-based matching (pixel equality for images, near-exact point-coordinate equality for LiDAR), not positional counting. - Missing calibration that has to be derived, never fabricated. semanticspray-plusplus ships with zero camera calibration anywhere in the source release, the devkit, or the community-supplied calibration file, so its camera stream is intentionally left unconnected to the transform tree rather than given an invented pose. oxford-spires-multimodal’s depth/normal-map pipeline needs open3d, which has no wheel for the build machine’s platform, so the devkit’s own math was reimplemented in NumPy/OpenCV instead of skipped. - Placeholder or leaked values that look like real per-point data but aren’t. canoe-multimodal’s LiDAR preallocates a 131,072-point array per frame; 70-93% of those points are (0,0,0) “no return” filler that has to be filtered out, or the 3D tile shows a dense blob at the origin. semanticspray-plusplus’s radar z-column is a dataset-wide constant that’s exactly the sensor’s calibrated mount height, leaked from upstream extraction rather than a real per-point measurement, so height is set via a static transform instead. Which MCAP dataset should you use? | Dataset | Domain | Episodes | Sensor Suite | Object Labels? | License | | --- | --- | --- | --- | --- | --- | | YUTO MMS | Ground-vehicle mobile mapping (SLAM) | 2 | Panoramic camera (Ladybug 5), tilted 32-beam LiDAR, GNSS/INS, IMU | None | Conflicting (site: CC BY-NC-SA; Zenodo: CC BY) | | Oxford Spires | Handheld backpack SLAM/reconstruction | 6 | 3× fisheye camera, 64-beam LiDAR, IMU, derived depth/normal maps | None | CC BY-NC-SA 4.0 | | CANOE | Aquatic uncrewed surface vessel (USV) navigation | 4 | Stereo camera, 360° radar, imaging sonar, 128-beam LiDAR, GNSS/INS, IMU | None | CC BY 4.0 | | FoMo | Off-road forest uncrewed ground vehicle (UGV) navigation | 6 | Stereo + mono camera, 2× LiDAR, FMCW radar, stereo audio, 2× IMU, post-processed kinematic (PPK)-GNSS | None | CC BY 4.0 | | TreeScope (VAT-0723) | uncrewed aerial vehicle (UAV) forestry LiDAR mapping | 10 | Stereo camera, 64-beam LiDAR, GPS, IMU | None | CC BY-NC-SA 4.0 | | TartanGround | Synthetic ground-robot navigation | 6 | Camera, segmentation stream, LiDAR, ego-pose, IMU | None | CC BY 4.0 | | SemanticSpray++ | Adverse-weather driving (vehicle spray) | 36 | Camera, 3× LiDAR, radar | 2D boxes, 3D boxes, LiDAR + radar point semantics (all 36 episodes) | CC BY 4.0 | | Boreas / Boreas-RT | Multi-season driving | 8 | Camera, 128-beam LiDAR (+ FMCW Aeva on Boreas-RT), 360° radar, GNSS/INS, IMU | 3D boxes (6 of 8 episodes only) | Unknown | | CMHT | Urban driving, sensor fusion | 4 | Camera, thermal/IR camera, LiDAR, mmWave radar, GPS/IMU | 3D boxes + derived 2D boxes (all 4 episodes) | CC0 1.0 | | NavWareSet | Indoor social navigation human-robot interaction (HRI) | 7 | Robot + external station: LiDAR ×2, camera ×2, AMCL pose, cmd_vel | Pedestrian 3D cuboids (SceneUpdate, not a label field) | CC BY 4.0 | All 10 MCAP datasets side by side: domain, episode count, sensor suite, whether real object labels are embedded, and license. Keep it honest. “Episodes” reflects each FiftyOne build’s current scope, not the full source dataset. Every one of these is a partial repackaging (see each domain section for exactly how much of the source is covered and why). What does "episode" mean in FiftyOne? For every dataset below, dataset.media_type == "multimodal" and one sample is one episode, one continuous recording, stored as a single .mcap file. There’s no per-frame image or point-cloud sample sitting alongside it; every camera frame, LiDAR scan, IMU sample, and transform inside that episode is decoded live by FiftyOne’s multimodal viewer, which renders Image, 3D, Map, Plot, Logs, and Message tiles in sync as you scrub through the recording: import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/boreas-multimodal") session = fo.launch_app(dataset) Object-level labels, where they exist, follow the same logic: a 3D box that’s only valid for one instant inside a 10-minute recording can’t live in a fixed-length sample field, so it’s logged as another timestamped channel inside the same MCAP (foxglove.SceneUpdate for 3D cuboids, foxglove.ImageAnnotations for 2D overlays), decoded in sync with the sensor data rather than stored separately. Dataset schema reference: media types, sample fields, and links | Dataset | Media Type | Notable Sample Fields | Label Fields | | --- | --- | --- | --- | | YUTO MMS | multimodal | sequence_id, route_length_m, collection_site, has_gps/has_imu/has_pointcloud | None | | Oxford Spires | multimodal | location, recording_date, has_rgb_lidar, has_depth, has_surface_normals, num_cameras | None | | CANOE | multimodal | route, radar_max_range_m, has_gps, has_imu (False, generic JSON, not schema-decoded) | None | | FoMo | multimodal | trajectory, deployment_date, gt_origin_utm, n_robosense_scans/n_leishen_scans, has_audio | None | | TreeScope | multimodal | site, session_id, platform, lidar_model, tags (manual_flight/autonomous_flight) | None | | TartanGround | multimodal | environment, trajectory, num_frames, duration | None | | SemanticSpray++ | multimodal | vehicle_type, ego_velocity_kmh, distance_to_object_m, has_object_labels (True for all 36) | 2D boxes, 3D boxes, point-wise semantics (native MCAP schemas, not FiftyOne label fields) | | Boreas / Boreas-RT | multimodal | episode_type (odometry/object_detection), dataset_variant, has_boxes, has_aeva | 3D boxes on 6 of 8 episodes | | CMHT | multimodal | drive_name, weather, lighting, is_preliminary_sync, has_3d_labels, has_2d_labels | 3D + derived 2D detections, all 4 | | NavWareSet | multimodal | scenario, robot_platform, behavior (social/non-social), has_pedestrian_annotations | Pedestrian cuboids (SceneUpdate) | What each dataset looks like once loaded in FiftyOne: media type, the sample fields you can filter on, and where labels actually live. Links: papers, datasets, and FiftyOne Spaces YUTO MMS - YUTO MMS: A comprehensive SLAM dataset for urban mobile mapping with tilted LiDAR and panoramic camera integration - YUTO MMS Hugging Face Dataset - YUTO MMS FiftyOne Space Oxford Spires - Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods - Oxford Spires Hugging Face Dataset - Oxford Spires FiftyOne Space CANOE - CANOE Hugging Face Dataset - CANOE FiftyOne Space FoMo - FoMo: A Multi-Season Dataset for Robot Navigation in Forêt Montmorency - FoMo Hugging Face Dataset - YUTO MMS FiftyOne Space TreeScope (VAT-0723) - TreeScope: An Agricultural Robotics Dataset for LiDAR-Based Mapping of Trees in Forests and Orchards - TreeScope Hugging Face Dataset - TreeScope FiftyOne Space TartanGround - TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation - TartanGround Hugging Face Dataset SemanticSpray++ - SemanticSpray++: A Multimodal Dataset for Autonomous Driving in Wet Surface Conditions - Energy-based Detection of Adverse Weather Effects in LiDAR Data - SemanticSpray++ Hugging Face Dataset - SemanticSpray++ FiftyOne Space Boreas / Boreas-RT CMHT - Boreas: A Multi-Season Autonomous Driving Dataset - Boreas Road Trip: A Multi-Sensor Autonomous Driving Dataset on Challenging Roads - Boreas Hugging Face Dataset - Boreas FiftyOne Space CMHT - CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving - CMHT Hugging Face Dataset - CMHT MMS FiftyOne Space NavWareSet - NavWareSet: A dataset of socially compliant and non-compliant robot navigation - NavWareSet Hugging Face Dataset Ground-vehicle and campus SLAM datasets (raw, no object labels) Both of these are raw multi-sensor recordings built for SLAM/odometry/reconstruction benchmarking, not perception. Neither carries a single object-level annotation. Their “ground truth” is a survey-grade trajectory, not a bounding box. YUTO MMS !YUTO MMS MCAP episode in FiftyOne showing an accumulated RGB-colorized LiDAR map, a 360-degree Ladybug panorama, and the GPS track around the Teledyne Optech campus. YUTO MMS MCAP episode in FiftyOne showing an accumulated RGB-colorized LiDAR map, a 360-degree Ladybug panorama, and the GPS track around the Teledyne Optech campus. YUTO MMS (York University Teledyne Optech Mobile Mapping System) is a SLAM benchmark collected with Teledyne Optech’s Maverick mobile mapping unit: a 360° Ladybug 5 panoramic camera (six stitched CCD sensors, 8000×4000), a Velodyne HDL-32E LiDAR tilted 45° off the camera axis, and a NovAtel GNSS/INS, recorded across a controlled parking-lot sequence in Vaughan, Ontario and a residential-loop sequence around York University’s Keele Campus in Toronto. This build covers 2 of the source’s 4 sequences (A and D); B and C are still in progress, since each requires re-tuning the accumulated-map voxel size and update rate per sequence rather than reusing a previous one’s settings. Every LiDAR point is colorized with real RGB sampled from the nearest-in-time panorama via the dataset’s own boresight calibration, not a synthetic colormap. Fields. sequence_id, route_length_m, collection_site, duration_s, plus has_image/has_pointcloud/has_gps/has_imu capability flags. No label fields exist, and none should. The source paper’s entire “Dataset” section covers only raw sensor streams and a photogrammetric ground-truth trajectory. Keep it honest. The license is genuinely conflicting: the project site and README both state CC BY-NC-SA 4.0, but every live Zenodo record for every sequence checked directly serves CC BY 4.0 via its own rel="license" header. Verify current status before redistributing. Dataset · FiftyOne Space Oxford Spires !FiftyOne multimodal viewer playing an Oxford Spires MCAP episode, with RGB-colorized LiDAR of the Bodleian Library and three fisheye camera frustums in the 3D tile. FiftyOne multimodal viewer playing an Oxford Spires MCAP episode, with RGB-colorized LiDAR of the Bodleian Library and three fisheye camera frustums in the 3D tile. Oxford Spires is the Oxford Robotics Institute’s large-scale LiDAR-visual localization, reconstruction, and radiance-field benchmark, captured by walking a handheld backpack unit called Frontier: three synchronized fisheye cameras (forward/left/right), a 64-beam LiDAR, and an IMU, through six historic Oxford landmarks (Bodleian Library, Christ Church, Keble College, New College, Blenheim Palace, Radcliffe Observatory Quarter). Each site also has a millimeter-accurate Terrestrial LiDAR Scanner reference model, used to produce centimeter-accurate ground-truth trajectories via iterative closest point (ICP) registration. Six of the source’s 24 sequences make it into this build, one per site, choosing the smallest available sequence at each landmark. Alongside the raw streams, every episode also carries the source devkit’s own derived depth maps, HSV depth overlays, and surface-normal maps per camera, so the devkit’s canonical visualization is reproducible directly in the App. Fields. location, recording_date, run_number, plus has_rgb_lidar/has_depth/has_surface_normals/num_cameras capability flags. No object-level labels. The source dataset’s “ground truth” is millimeter-accurate TLS 3D models and centimeter-accurate trajectories, not per-sample annotations. Good to know. Why is the Map tile empty for every episode? There’s no GNSS/GPS stream anywhere in the source dataset, so the App’s Map tile has nothing to show, by design, not a parsing gap. Dataset · FiftyOne Space Aquatic and off-road robotics datasets (raw, no object labels) Same story as the ground-SLAM group: raw sensor streams plus a post-processed ground-truth pose, no object annotations. But the environments couldn’t be more different: a lake, a boreal forest, and a forest canopy from above. CANOE !Height-colored LiDAR point cloud of a lake shoreline and tree line from a CANOE MCAP episode in FiftyOne's 3D tile. Height-colored LiDAR point cloud of a lake shoreline and tree line from a CANOE MCAP episode in FiftyOne's 3D tile. CANOE (Canadian Aquatic Navigation for Observation of the Environment) is ASRL’s marine navigation dataset, collected on an uncrewed surface vessel traversing a lake and reservoir in Ontario over three days. The sensor suite is genuinely unusual for a robotics dataset: a 360° Navtech scanning radar (dual range modes: 1000 m long-range or 150 m Doppler), a 128-beam Ouster LiDAR, a stereo camera, an Oculus imaging sonar, dual Starfish sidescan sonar, and a NovAtel GNSS/INS post-processed to 2-4 cm RMS accuracy. Four of the source’s 8 public sequences are included here, chosen for sensor/environment diversity: three fixed-duration windows plus the officially curated “Sample” sequence in full. Fields. route, radar_max_range_m, source, plus has_image/has_pointcloud/has_gps capability flags. has_imu is False for every episode in this build even though an /imu topic exists, because it rides on a generic JSON channel rather than a schema-decoded one. Keep it honest. Sidescan sonar is out of scope for this build. sidescan.xtf/sidescan.csv aggregate an entire sequence’s pings into single files with no per-frame breakdown, so they were never decoded into any MCAP channel. Dataset · FiftyOne Space FoMo !FoMo MCAP episode in FiftyOne showing the Navtech radar bird's-eye-view stream next to a colorized forest LiDAR point cloud. FoMo MCAP episode in FiftyOne showing the Navtech radar bird's-eye-view stream next to a colorized forest LiDAR point cloud. FoMo is a year-long robotic data collection recorded in Forêt Montmorency, a boreal forest 80 km north of Quebec City, using a Clearpath Warthog UGV carrying two LiDARs (RoboSense + Leishen), an frequency-modulated continuous wave (FMCW) radar, a stereo + mono camera pair, two IMUs, stereo microphones, and PPK-GNSS ground truth. The full source is 9.43 TB across 60 sessions; this build samples 6 episodes, one per trajectory type, across 6 deployments spanning winter (-19°C, over a meter of snow) through summer conditions, kept under a 30 GB budget. Fields. trajectory, trajectory_description, deployment_date, gt_origin_utm, sensor-specific frame counts (n_robosense_scans, n_leishen_scans, n_navtech_frames, n_audio_left_clips), plus has_image/has_pointcloud/has_imu/has_gt_pose/has_audio flags. Keep it honest. This is a 6-of-60-session recon sample under a size budget, not a statistically representative subset. Don’t use it as a stand-in for the full dataset’s scale in a SLAM/odometry benchmark. Dataset · FiftyOne Space TreeScope (VAT-0723) !TreeScope VAT-0723 MCAP episode in FiftyOne with UAV LiDAR point clouds, two stereo camera tiles, an odometry plot, and the Map tile. TreeScope VAT-0723 MCAP episode in FiftyOne with UAV LiDAR point clouds, two stereo camera tiles, an odometry plot, and the Map tile. TreeScope is a LiDAR dataset for precision agriculture and forestry, collected across six forest and orchard sites with UAV and mobile-robot platforms. This build covers 10 episodes from just one site, VAT-0723 (Appomattox-Buckingham State Forest, Virginia), the only confirmed TreeScope site with released GPS data, making FiftyOne’s Map tile achievable at all. The UAV platform (a 4.2 kg Falcon 4) carries an Ouster OS1-64 LiDAR, stereo cameras, a VectorNav IMU, and a UBlox GPS. Each episode merges that session’s raw and Faster-LIO-processed ROS bag pair, remapping the processed trajectory to /refined/Odometry so both the onboard and offline-refined estimates stay distinguishable. Fields. site, session_id, platform, lidar_model, flight-mode tags (manual_flight/autonomous_flight), plus has_image/has_pointcloud/has_gps/has_imu/has_logs flags. Keep it honest. No ground truth is attached to any of these 10 episodes, deliberately. TreeScope’s site-level semantic-segmentation labels exist but their labeled-frame timestamps fall entirely outside every chosen episode’s flight window. metadata/VAT-0723.json independently confirms no semantic labels or diameter at breast height (DBH) measurements for these sessions. This is also a non-commercial dataset (CC BY-NC-SA 4.0). Dataset Synthetic ground-robot navigation TartanGround !TartanGround synthetic MCAP episode in FiftyOne showing a night city camera stream, its paired segmentation stream, and LiDAR playing in sync. TartanGround synthetic MCAP episode in FiftyOne showing a night city camera stream, its paired segmentation stream, and LiDAR playing in sync. TartanGround is CMU AirLab’s large-scale synthetic dataset for ground-robot perception and navigation, generated across a range of simulated environments. This build covers 6 trajectories, one per environment (AbandonedFactory, CyberPunkDowntown, GreatMarsh, Hospital, JapaneseCity, NordicHarbor), each converted to a native multimodal MCAP episode carrying a front camera, its paired segmentation stream, per-frame LiDAR point clouds, ego pose, and IMU telemetry on a shared 10 Hz frame clock. Fields. environment, trajectory, num_frames, duration. Streams: /front-camera (JPEG), /front-segmentation (PNG), /lidar, /ego-pose, /imu.plot. Good to know. Unlike every other raw dataset in this post, TartanGround does ship a segmentation stream (/front-segmentation), but like the object-detection boxes elsewhere, it’s a synchronized MCAP channel decoded live in the Image tile, not a FiftyOne Segmentation label field. Dataset Datasets with real embedded object detections These three are the exception to everything above: real, dense, per-frame object annotations, logged as additional timestamped channels inside the same MCAP timeline as the sensor data, decoded live by the multimodal viewer’s Image and 3D tiles in sync with playback, rather than stored as separate FiftyOne label fields. SemanticSpray++ !SemanticSpray++ MCAP episode in FiftyOne with a 3D LiDAR box on the lead van and point-wise spray semantics colorized in the point cloud. SemanticSpray++ MCAP episode in FiftyOne with a 3D LiDAR box on the lead van and point-wise spray semantics colorized in the point cloud. SemanticSpray++ labels a 36-scene subset of the RoadSpray dataset: an ego vehicle following a lead vehicle (a small car or a large van) on wet pavement at matched speeds from 50-130 km/h, generating a trailing water-spray plume off the wet surface. Every one of the 36 episodes in this build has the complete label set: 2D camera boxes, 3D LiDAR boxes, LiDAR point-wise semantics (background/foreground/noise), and radar point-wise semantics, all for the lead vehicle. Across the 36 episodes: 1,362 Car and 1,225 Van 2D/3D box instances, plus 77.8 million labeled LiDAR points. Fields. vehicle_type (Golf/Crafter), ego_velocity_kmh, distance_to_object_m, has_object_labels (True for all 36). Label channels: /camera/annotations (2D boxes), /objects/lidar_boxes (3D boxes), colorized /velodyne_points (point semantics), colorized radar points. Keep it honest. The other 171 scenes in the same source download (the earlier, LiDAR-only “SemanticSpray” release with no boxes) are deliberately not included. This build targets exactly the 36-scene, fully-labeled SemanticSpray++ subset. Dataset · FiftyOne Space Boreas / Boreas-RT !Boreas MCAP episode in FiftyOne showing 3D object cuboids over the LiDAR scene, with synchronized camera and radar tiles below. Boreas MCAP episode in FiftyOne showing 3D object cuboids over the LiDAR scene, with synchronized camera and radar tiles below. Boreas is ASRL’s multi-season driving dataset, collected by repeatedly driving a route near the University of Toronto over a full year: sun, rain, falling snow. Boreas Road Trip (Boreas-RT) extends it with 9 new, more diverse routes and adds an Aeva FMCW Doppler LiDAR plus a standalone IMU. A separate dedicated recording, boreas-objects-v1, provides 326,180 3D object annotations (Car/Pedestrian/Cyclist/Misc) across 7,111 labeled LiDAR frames. This build covers 2 driving episodes (one Boreas, one Boreas-RT, each a 10-second window) plus 6 object-detection windows carved from boreas-objects-v1. Fields. episode_type (odometry/object_detection), dataset_variant, has_boxes, has_aeva, has_dmu_imu. Label channels (object-detection episodes only): /boxes (3D SceneUpdate cuboids), /camera/boxes and /radar/boxes (2D wireframe overlays), all carrying label/uuid/numPoints metadata inspectable via FiftyOne’s click-to-inspect panel. Keep it honest. has_boxes is True for only 6 of the 8 episodes. The 2 driving/odometry episodes carry no object annotations at all. Only obj_train-split windows are included; obj_test ground truth is withheld by the source dataset for leaderboard fairness. Dataset · FiftyOne Space CMHT !CMHT night-rain MCAP episode in FiftyOne with car and truck boxes on the RGB camera, the same detections on the thermal IR stream, and 3D cuboids in the LiDAR scene. CMHT night-rain MCAP episode in FiftyOne with car and truck boxes on the RGB camera, the same detections on the thermal IR stream, and 3D cuboids in the LiDAR scene. The CMHT Autonomous Dataset is McMaster University’s multi-sensor driving dataset recorded around Hamilton, Ontario: a single rig combining a Velodyne LiDAR, an mmWave radar, a monocular RGB camera, and a FLIR thermal/IR camera, across four drives spanning dusk/clear and night/rain conditions. Over 9,000 frames are labeled with 3D position, size, rotation, class (Car/Truck/Van/Pedestrian/Bus/LongVehicle), and a persistent object ID. This repackaging re-embeds those separately-published labels back into each drive’s MCAP timeline, synced to the exact sensor message each was originally annotated from. Fields. drive_name, weather, lighting, is_preliminary_sync, has_3d_labels, has_2d_labels. Label channels: /Labels_3D (vision_msgs/Detection3DArray, one message per labeled LiDAR frame) and /Labels_2D_front//Labels_2D_ir (vision_msgs/Detection2DArray). Keep it honest. The 2D boxes are derived, not manually annotated. Only the 3D LiDAR-frame boxes were hand-labeled (via SUSTechPoints); the 2D camera and thermal boxes are projections of those same 3D boxes onto each image plane, computed by this repackaging using the authors’ own calibration and projection method. is_preliminary_sync is also True for 3 of the 4 drives: only the LiDAR triggers the cameras, while radar, GPS, and IMU are approximately, not frame-exactly, time-aligned. Dataset · FiftyOne Space Social navigation and human-robot interaction NavWareSet !NavWareSet Circular Crossing MCAP episode in FiftyOne showing the Jackal robot's camera view, the ground-station camera, and pedestrian and trajectory annotations in the 3D scene. NavWareSet Circular Crossing MCAP episode in FiftyOne showing the Jackal robot's camera view, the ground-station camera, and pedestrian and trajectory annotations in the 3D scene. NavWareSet is a multimodal dataset of socially-compliant and non-compliant robot navigation around human pedestrians, recorded in a controlled indoor environment across seven canonical social-navigation scenarios (Frontal Approach, Pedestrian Obstruction, Blind Corner, Following Human, Perpendicular Traffic, Circular Crossing, Object Handover). Every scene is captured simultaneously by the robot’s own sensors and by a stationary external Ground-truth Recording Station (a LiDAR mounted above a camera), which also provides the point cloud human trajectories were manually annotated on. Seven of the source’s 48 scenes are included here, Clearpath Jackal platform only. The Toyota HSR platform isn’t imported yet, though the merge pipeline already supports it. Fields. scenario, robot_platform, behavior (social/non-social), participant_group, has_pedestrian_annotations. Label channel: /annotations/pedestrians (foxglove.SceneUpdate, persistent per-person cuboids carrying the source’s own tracking ID). Good to know. Why isn’t there a Detections field for the pedestrian boxes? Pedestrian cuboids are a time-synced channel meant to play back alongside the LiDAR/camera channels in the 3D tile: SceneUpdate entities carry a persistent entity ID across frames, matching the source annotation format’s per-person UUID directly, with no separate tracking field needed. Dataset Where this is all headed: physical AI Across all 10 datasets, the actual training bottleneck is real, timestamped, multi-sensor recordings of something moving through the world (a car, a boat, a UAV, a robot navigating around a person) with enough synchronized modalities to reconstruct what happened. That’s the raw material world models and embodied agents train on, and it’s the shape of data MCAP was built to hold and FiftyOne’s multimodal viewer was built to make explorable without a custom bag-parsing script. Try it yourself pip install -U fiftyone import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/semanticspray-plusplus") # or any dataset above session = fo.launch_app(dataset) - FiftyOne on GitHub - Voxel51 on Hugging Face - MCAP format specification Frequently Asked Questions What is an MCAP file? MCAP is a modular, serialization-agnostic binary container format for timestamped pub/sub data: one file holds any number of named channels (camera, LiDAR, IMU, GPS, transforms), each indexed for fast seeking. It’s been the default log format for ROS 2 since the Iron release. Can I view ROS bag data in FiftyOne? Not directly. ROS 1 .bag and ROS 2 .db3 files need to be converted to .mcap first (via the mcap CLI, rosbags, or a custom script), at which point FiftyOne’s media_type: "multimodal" support decodes them automatically based on the .mcap file extension. Which of these datasets actually have object detection labels? Only three: semanticspray-plusplus (all 36 episodes, 2D boxes, 3D boxes, point-wise semantics), boreas-multimodal (6 of its 8 episodes, via the has_boxes field), and cmht-autonomous-driving (3D boxes plus derived 2D projections, all 4 episodes). navwareset has real pedestrian cuboids but as a SceneUpdate channel, not a FiftyOne label field. The remaining 6 are raw sensor data with no object annotations at all. What’s the difference between a FiftyOne multimodal dataset and a grouped dataset? A multimodal dataset (what every dataset in this post is) has media_type: "multimodal" and one sample per .mcap episode, with every stream inside decoded live by the viewer’s tiles. A grouped dataset instead has media_type: "group" and multiple related samples (e.g., different camera slices, a 3D scene) linked under one group field, a different mechanism used when labels need to attach as ordinary per-frame FiftyOne fields rather than live inside an MCAP timeline. Which MCAP datasets can I use commercially? Check each license individually. semanticspray-plusplus, fomo-multimodal-sample, canoe-multimodal, navwareset, and TartanGround are CC BY 4.0; cmht-autonomous-driving is CC0 1.0. oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA 4.0 (non-commercial only). boreas-multimodal’s license is unlisted. yuto-mms-multimodal has a genuine conflict between its project site (CC BY-NC-SA) and its live Zenodo records (CC BY). Verify directly before relying on either.

Data Curation, the Voxel51 Agent, and In-App Annotation: A CropAndWeed Walkthrough A beat-by-beat walkthrough of CropAndWeed, verified with the FiftyOne SDK — including the moment I caught myself about to say something that wasn't quite true. !The CropAndWeed dataset open in the FiftyOne App, showing the sample grid, a 3D embeddings plot colored by class, and a model evaluation dashboard. The CropAndWeed dataset open in the FiftyOne App, showing the sample grid, a 3D embeddings plot colored by class, and a model evaluation dashboard. CropAndWeed is a hand-labeled dataset comprised of 111,953 crop and weed instances across 8,034 field images. On paper, it looks clean. In practice, it isn't. Class imbalance, near-duplicate images, and a handful of genuinely broken labels are all hiding in there, and I wanted to find them without writing a wall of analysis code by hand. So I opened the dataset in the FiftyOne App and let the Voxel51 Agent narrate its way through the mess for me. Mostly hands-off, mostly no terminal. That audit turned out to be just the setup. CropAndWeed also has real rare classes buried in it, like species with as few as one labeled instance, that made it a good candidate for something more ambitious: an actual rare-class mining funnel. Compress, embed across three different foundation models, mine, and turn the output into an annotation-triage queue that a human could actually act on. Steps 1 through 6 below cover the data-centric audit. Steps 7 through 12 cover the mining funnel, including the two moments where the agent told me something that sounded almost too convenient, and I had to go verify it myself before I believed it. Key Takeaways - CropAndWeed's class imbalance ranges from 33,000 instances down to 1. But the top "class," Vegetation, isn't a species. It's the dataset's own fallback bucket (label_id 255) for anything too tiny or ambiguous to classify. Exclude it, and the corrected species-level ratio is 12,132 to 1: Green bristlegrass (12,132) vs. Black horehound (1). - The dataset's capture protocol deliberately spaced images within a session by at least 3 meters to avoid redundancy. fiftyone.brain.compute_uniqueness shows it only partially worked: the 50 least-unique samples across the entire 8,034-image dataset are concentrated in just 16 of its 929 recording sessions, not evenly distributed. One session alone (Application Set 0067) accounts for 14 of those 50. - Using a purely geometric check, each detection's bounding_box and its stem_x/stem_y attributes, uncovered 370 degenerate (near-zero-area) boxes across 192 samples, and 23 detections whose stem point falls outside their bounding box. Those were annotation defects, caught without running a single model. - 234 of the 8,034 images (2.9%) ship with a segmentation mask but zero ground_truth detections. Worth knowing before anyone trains a detector off this dataset without checking first. - Whole-image embedding search cannot find a rare class hiding in a busy scene. Not with CLIP, not with Qwen3-VL-Embedding-2B on a text prompt, not with an unsupervised outlier check on CRADIOv4 or Qwen3-VL. All three foundation models, all three query mechanisms, zero of eight held-out Black nightshade samples recovered. Patch-level embedding search is the actual fix, but it comes with a performance ceiling and isn't a magic bullet. - Bounding-box area is a nearly-free triage signal that needed zero new model compute: the 23 known stem-point defects average 1,423.9 px², roughly 10x smaller than the dataset-wide mean of 13,681.3 px². - The tempting rule "rare classes hide in the Experimental Set" only holds for 6 of 19 rare classes checked — 13 skew the other way. A pattern that fits one example isn't a rule until it's checked against more than one example. - Every number in this post came from either a sentence typed into the Voxel51 Agent or a script short enough to read in one sitting (scripts/dataset_facts.py) — no separate analysis notebook, no manual counting in a spreadsheet. Meet CropAndWeed, Which Looks Cleaner Than It Is CropAndWeed is developed by the AIT Austrian Institute of Technology (Steininger, Trondl, Croonen, Simon, and Widhalm; WACV 2023) and is built for automated, plant-specific weed intervention. Images come from two sources: an "Application Set" shot at real commercial farm sites in Austria under uncontrolled field conditions, and an "Experimental Set" shot on dedicated plots grown specifically to enrich the dataset with rare weed species that don't show up often enough in the wild. Both were captured over four years, March through July, with a fixed 50mm lens from roughly 1.1 meters up, in a top-down view. The FiftyOne build I'm working with has three label types living on every sample: ground_truth (Detections, one box per plant instance), stem_points (Keypoints, one per instance marking its stem position), and segmentation (a per-pixel semantic mask). Four string primitives (moisture, soil, lighting, separability) record the conditions at the time of capture. > Keep it honest. The official taxonomy has 100 classes (99 species/growth-stage labels plus the Vegetation fallback). Only 84 of them actually appear in ground_truth.detections.label in this particular 8,034-image release. The other 16 exist in the spec but not in this specific set of annotated images. And there's no official train/val/test split shipped with the raw annotation release; whatever curated subset comes out of this workflow is one I build myself, not one that already existed. Step 1: Ask the Agent What's Actually in Here Instead of writing out a description of the dataset myself, I open the FiftyOne App with CropAndWeed loaded and type into the Agent panel: > "Give me a quick overview of this dataset — media type, label fields, and anything unusual in the metadata." The agent comes back narrating the three label types, the four condition primitives, and the session structure. That's the whole point of leading with the agent instead of writing the summary myself: you hear the dataset describe itself, and nobody had to write dataset.get_field_schema() by hand to make it happen. !The Voxel51 Agent answering a prompt for a dataset overview by running a dataset_summary tool over CropAndWeed's media type and label fields. The Voxel51 Agent answering a prompt for a dataset overview by running a dataset_summary tool over CropAndWeed's media type and label fields. | Issue | What was found | How it was found | | --- | --- | --- | | Class imbalance (corrected) | 12,132:1 species-level ratio (Green bristlegrass vs. Black horehound), after excluding the Vegetation fallback bucket | count_values on ground_truth.detections.label | | Near-duplicate concentration | 50 least-unique images cluster in 16 of 929 sessions (one session accounts for 14) | fiftyone.brain.compute_uniqueness | | Degenerate boxes | 370 near-zero-area boxes across 192 samples | Geometric check on bounding_box | | Stem points outside their box | 23 detections | Geometric check on stem_x / stem_y | | Masks with no detections | 234 of 8,034 images (2.9%) | Field-presence check | Data-quality issues surfaced in CropAndWeed during a data-centric audit, each found in the FiftyOne App without training a model. Step 2: The 33,000:1 Reveal — and Why That Number Lies a Little Next: > "Show me the distribution of ground_truth.detections.label as a chart, sorted descending." !The Voxel51 Agent generating a descending class-distribution chart of CropAndWeed detection labels in the Histograms panel. The Voxel51 Agent generating a descending class-distribution chart of CropAndWeed detection labels in the Histograms panel. The chart is dramatic. Vegetation sits at 33,000 instances. Black horehound, Frosted orach, and Common dandelion sit at 1 apiece. That's a genuine 33,000:1 ratio, and it's tempting to just land on that number and move on. > Good to know. What actually is Vegetation? It's not a plant. Per the dataset's own documentation, Vegetation is label_id 255. It's a fallback class assigned to any instance too small (under roughly 16x16 pixels), too ambiguous, or otherwise excluded from fine-grained classification. It shows up in both ground_truth and segmentation. In other words, the single most common "class" in this dataset is a data-quality signal wearing a species label. Pull it out, and the real species-level imbalance is Green bristlegrass at 12,132 instances vs. Black horehound at 1, still more than 12,000 to 1, and now it's a claim about actual biology instead of an artifact of how the annotation pipeline handles hard cases. I'll lead with 33,000:1 for the shock value, then immediately hand you the correction, because the correction is the more useful lesson: check what's actually inside your biggest bucket before you build a dashboard, a report, or a rebalancing strategy around it. !A FiftyOne dashboard showing CropAndWeed's class imbalance, with the Vegetation fallback bucket dwarfing every species, alongside soil moisture and soil type distributions. A FiftyOne dashboard showing CropAndWeed's class imbalance, with the Vegetation fallback bucket dwarfing every species, alongside soil moisture and soil type distributions. Step 3: The Sampling Protocol vs. the Embedding Model Next on the list of messy realities: near-duplicate frames. I almost got this one wrong because I didn’t fully understand what one of the sample fields meant. I initially thought that image_id was a unique value for each image. But really, it's a field that repeats hundreds of times throughout the dataset (0002 appears 519 times, 0004 appears 505 times). That repetition looked like duplicate frames to me. It's a satisfying idea: drone-style top-down field imagery, shot in bursts, probably has redundant frames, right? !The Voxel51 Agent explaining that CropAndWeed's image_id repeats per recording session and is not a reliable signal for duplicate images. The Voxel51 Agent explaining that CropAndWeed's image_id repeats per recording session and is not a reliable signal for duplicate images. > Keep it honest. image_id is a 4-digit counter that resets to 0000 at the start of every session. Its repetition across the dataset is just a filename convention. Every session numbers its own images starting from zero, so, of course, the string "0002" appears hundreds of times across the dataset's 929 recording sessions. It says nothing about duplicate frames. (That 929 number took its own correction, too: a naive dataset.distinct("session") returns only 801, because 128 session-ID strings are also reused across the two recording sets — the same "0001" exists in both. Group by (recording_set, session) instead and you get the 929 the dataset card actually reports.) Worse, the card explicitly states that images within a session were captured at least 3 meters apart specifically to avoid redundancy. My hunch wasn't just unverified; it was aimed at a field designed to disprove it. So I asked the actual question instead: did that 3-meter rule work, as far as an embedding model is concerned? I ran fiftyone.brain.compute_uniqueness on the full dataset and sorted for the least-unique samples: import fiftyone as fo import fiftyone.brain as fob dataset = fo.load_dataset("CropAndWeed") fob.compute_uniqueness(dataset) least_unique = dataset.sort_by("uniqueness")[:50] print(least_unique.count_values("session")) The 50 least-unique samples across the entire 8,034-image dataset are concentrated in just 16 of its 929 recording sessions — one session (Application Set 0067) alone accounts for 14 of those 50 samples. If uniqueness were spread evenly across sessions, you'd expect something closer to 50 different sessions represented, not 16. The spacing rule didn't fail outright, but it didn't fully outrun how similar one field, photographed patch by patch, looks to a generic embedding model. That's a more honest and, I think, more interesting story than "these are duplicates" — it's a story about the limits of a sampling protocol, not a story about sloppy data collection. !The Voxel51 Agent surfacing the 50 least-unique CropAndWeed samples with compute_uniqueness, shown on an embeddings plot colored by uniqueness score. The Voxel51 Agent surfacing the 50 least-unique CropAndWeed samples with compute_uniqueness, shown on an embeddings plot colored by uniqueness score. Step 4: Auditing Labels Without a Model Every one of the 111,953 instances in ground_truth was hand-drawn, then reviewed by majority vote among annotators for the hardest image batches, per the dataset's own annotation process. That's about as close to "trustworthy" as a label gets. I still checked. Two checks, neither requiring a model or predictions of any kind: Degenerate boxes. Any detection with a bounding box side under 0.001 (normalized) is functionally a point, not a box: from fiftyone import ViewField as F tiny = dataset.filter_labels( "ground_truth", (F("bounding_box")[2] < 0.001) | (F("bounding_box")[3] < 0.001), ) 370 instances, across 192 samples. !The Voxel51 Agent auditing CropAndWeed bounding-box areas to flag degenerate near-zero-area boxes, with a bbox area histogram and label distribution. The Voxel51 Agent auditing CropAndWeed bounding-box areas to flag degenerate near-zero-area boxes, with a bbox area histogram and label distribution. Stem points outside their own box. Each Detection in ground_truth carries its own stem_x/stem_y attributes in pixel coordinates — so checking whether a plant's marked stem position actually falls inside its own bounding box doesn't require matching across the separate stem_points field at all, just comparing a detection against itself: for sample in dataset.select_fields(["ground_truth", "metadata"]): width, height = sample.metadata.width, sample.metadata.height for det in sample.ground_truth.detections: x, y, w, h = det.bounding_box box_x0, box_y0 = x * width, y * height box_x1, box_y1 = (x + w) * width, (y + h) * height if not (box_x0 <= det.stem_x <= box_x1 and box_y0 <= det.stem_y <= box_y1): # flag it ... 23 detections fail that check. One of them is a Sugar beet six-leaf stage instance — a real crop, not the Vegetation catch-all — which makes it a better on-screen example than a generic fallback label. I bookmarked that exact sample (6a4ebad92c05998791d538a9, detection 6a4ebad82c05998791d5002f) as a saved view, so I always have a reliable example on hand even if a live filter surfaces something different in the moment. !The Voxel51 Agent running a geometric check for stem points that fall outside their own bounding box and tagging the offending detections for reannotation. The Voxel51 Agent running a geometric check for stem points that fall outside their own bounding box and tagging the offending detections for reannotation. Step 5: Fix a Label Live in the App, No Code This is the part the rest of the workflow was building toward: opening that bookmarked sample in the App's expanded view, switching to the Annotate tab, and fixing the stem point by dragging it into the actual bounding box — no code, no notebook, just the new in-App annotation editor. From there, the tour extends to the other label types on the same dataset: resizing a ground_truth box, adding an instance mask to a detection via the AI-assist tool (click one positive point, get a mask), and editing a moisture or lighting value through the primitive dropdown editor. Three label types plus four editable primitives, all on one dataset, is most of what the in-App annotation surface can do, in one sample. > Keep it honest. The AI-assist mask tool is the one step in this whole workflow I haven't dry-run yet. Everything else in this post is backed by a script I actually ran against the real data. This one still needs a hands-on pass in the App itself before I'd claim it works exactly as described. !Fixing a mislabeled stem point live in the FiftyOne App's in-app annotation editor by dragging it inside the plant's bounding box. Fixing a mislabeled stem point live in the FiftyOne App's in-app annotation editor by dragging it inside the plant's bounding box. Step 6: Export a Curated Set That Didn't Exist Before No train/val/test split ships with CropAndWeed's raw annotation release, and 234 of the 8,034 images (2.9%) have a segmentation mask but zero ground_truth detections — meaning a naive "just train a detector on everything" approach would silently include images with nothing to detect. Tag the quality-checked, deduplication-aware subset, filter to it, and export — that's the moment the whole exploration turns into something you could actually retrain on, and it's also the first split this dataset has ever had, because nobody shipped one. Step 7: Mining a Rare Class That's Already Hiding in Plain Sight Black nightshade has 14 instances across 9 images in CropAndWeed (a real rare class). I treated it like an unlabeled pool: pretend I only know about one image that contains it, then try to find the rest using embedding search. import fiftyone.brain as fob from fiftyone import ViewField as F seed_view = dataset.filter_labels("ground_truth", F("label") == "Black nightshade") seed_id = sorted(seed_view.values("id"))[0] fob.compute_similarity(dataset, model="clip-vit-base32-torch", brain_key="clip_sim") results = dataset.sort_by_similarity(seed_id, k=200, brain_key="clip_sim") > Keep it honest. Zero of 8 held-out true positives recovered, even in the top 200. I tried to explain it away as a CLIP problem — so I re-ran the same search with Qwen3-VL-Embedding-2B, seeded with plain-language text instead of an image (five phrasings, including the scientific name Solanum nigrum). Zero, every time. Then I checked whether an entirely unsupervised approach — CRADIOv4's and Qwen3-VL's own representativeness and uniqueness scores, no seed of any kind — flagged it as an outlier. Also zero. Three foundation models, three different query mechanisms, all zero. That's not bad luck — a whole-image embedding represents the scene, not the one small plant sitting among 37 other detections in the frame. !Whole-image embedding similarity searches in FiftyOne failing to recover held-out Black nightshade samples across CLIP and CRADIOv4 embeddings. Whole-image embedding similarity searches in FiftyOne failing to recover held-out Black nightshade samples across CLIP and CRADIOv4 embeddings. Step 8: Going to the Patch Level — the Fix, and Its Honest Ceiling The fix is architectural, not a better prompt: embed individual plant crops instead of whole images. fob.compute_similarity( pool, patches_field="ground_truth", model="clip-vit-base32-torch", brain_key="clip_patch_sim", ) patches_view = pool.to_patches("ground_truth") results = patches_view.sort_by_similarity(seed_patch_id, k=200, brain_key="clip_patch_sim") 2 of 8 held-out positives recovered in the top-200 — a real ~15x enrichment over the ~0.13 expected by chance. Adding two more seed patches (three total) recovers 2 of the remaining 6. That's a genuine improvement, and a genuine ceiling in the same breath: a handful of seeds can't cover every visual mode of a rare species — growth stage, lighting, occlusion — which is exactly why a real mining funnel needs "confirm" and "verify coverage" as their own steps, not an afterthought. A Second Foundation Model (CRADIOv4), and a Heatmap Diagnostic While I had embeddings loaded, I also brought in NVIDIA's CRADIOv4-H, a vision foundation model distilled from three teachers (SigLIP2, DINOv3, SAM3) that ships a spatial output mode FiftyOne renders as a Heatmap — a live, qualitative check of whether a model's attention agrees with where a human actually drew a box. import fiftyone.zoo as foz foz.register_zoo_model_source("https://github.com/harpreetsahota204/CRADIOv4") model = foz.load_zoo_model("nv_labs/c-radio_v4-h", output_type="spatial") dataset.apply_model(model, label_field="cradio_heatmap") > Good to know. CRADIOv4-H embeddings are 2560-dim; Qwen3-VL-Embedding-2B embeddings are 2048-dim. Computing embeddings, similarity, uniqueness, representativeness, a 3D UMAP visualization, and — for CRADIOv4 — a spatial heatmap, across all 8,034 images for both models, is roughly 53 minutes combined on a GPU. A one-time cost, not something you'd want to wait on interactively. !NVIDIA CRADIOv4-H rendered as a spatial heatmap in FiftyOne, showing where the foundation model concentrates attention on a plant instance. NVIDIA CRADIOv4-H rendered as a spatial heatmap in FiftyOne, showing where the foundation model concentrates attention on a plant instance. Step 10: Area — the Signal That Was Free All Along Before spending any embedding compute at all, there's a nearly-free signal sitting in the boxes I already had: size. bbox_area_expr = ( F("$metadata.width") * F("bounding_box")[2] * F("$metadata.height") * F("bounding_box")[3] ) dataset.set_field("ground_truth.detections.bbox_area_px", bbox_area_expr).save() The 23 known out-of-bounds stem-point defects from Step 4 have a mean bbox_area_px of 1,423.9 px² — roughly 10x smaller than the dataset-wide mean of 13,681.3 px². Small area is a real, cheap proxy for "this label probably deserves a second look," and it costs nothing beyond a field I'd already computed. Step 11: The Free Signal That Wasn't as Free as It Looked Black nightshade is 8 of its 9 samples in the Experimental Set — the plots the dataset's authors cultivated specifically to enrich rare species. Tempting rule: check the enrichment set first when hunting rare classes. > Keep it honest. Checked across all 19 classes with 40 or fewer instances: only 6 actually skew toward Experimental Set. Thirteen skew the other direction. Generalizing a triage rule from the one example that happened to fit the story would have meant telling you something wrong. Step 12: Building an Actual Triage Queue — and Correcting a Second Assumption I expected CRADIOv4's and Qwen3-VL's representativeness and uniqueness scores to flag rare-species candidates the way the mining funnel was supposed to. Checked directly against the real Black nightshade samples: zero overlap. What the "outlier" tier actually contains, inspected directly, is images with an unusually high density of common classes — Meadow-grass shows up 96 times across just 17 images. Busy, cluttered scenes, not rare-species scenes — a real, useful signal (plausibly the hardest images to annotate correctly), just not the one I went looking for. Combined with bounding-box area, that becomes a real, tag-based annotation-triage queue: tier1_ids = cross_model_outlier_ids & (small_bbox_area_ids | sparse_detection_ids) dataset.select(list(tier1_ids)).tag_samples("triage_tier_1") 17 images land in tier 1 (review first), 8,001 at normal priority, 16 safe to skip. !The Voxel51 Agent cross-checking outlier and uniqueness scores against real Black nightshade samples while building an annotation triage queue. The Voxel51 Agent cross-checking outlier and uniqueness scores against real Black nightshade samples while building an annotation triage queue. Try It Yourself - agentic_annotation on GitHub (harpreetsahota204/agentic_annotation) — the full, runnable code for everything in this post: every script referenced here, the rare-class mining funnel end to end, and the triage-queue builder. Start there if you want to run this yourself rather than just read about it. - Voxel51/CropAndWeed on Hugging Face (https://huggingface.co/datasets/Voxel51/CropAndWeed) — the exact dataset used in this post - CropAndWeed dataset repository (https://github.com/cropandweed/cropandweed-dataset) — the original annotation release and label-remapping scripts - Steininger, D., Trondl, A., Croonen, G., Simon, J., & Widhalm, V. (2023). The CropAndWeed Dataset: A Multi-Modal Learning Approach for Efficient Crop and Weed Manipulation. WACV 2023. - CRADIOv4 (https://github.com/harpreetsahota204/CRADIOv4) — the FiftyOne zoo model source used for embeddings and spatial heatmaps in steps 7-12 - Qwen3-VL-Embedding-2B model card (https://docs.voxel51.com/model_zoo/models/qwen3_vl_embedding_2b_torch.html) — the text-and-image embedding model used alongside CRADIOv4 - FiftyOne (https://github.com/voxel51/fiftyone) — pip install -U fiftyone - scripts/dataset_facts.py — the verification script behind every number in steps 1-6 - scripts/rare_class_mining.py, rare_class_mining_patches.py, rare_class_mining_multiseed.py, compute_foundation_embeddings.py, compute_areas.py, recording_set_signal.py, build_triage_queue.py — the scripts behind every number in steps 7-12 Frequently Asked Questions Is Vegetation a weed species in CropAndWeed? No. It's label_id 255, the dataset's explicit fallback class for plant instances too small (under roughly 16x16 pixels), too ambiguous, or otherwise excluded from fine-grained species classification. It's the most common label in ground_truth.detections.label (33,000 instances) precisely because it's a catch-all, not because one species dominates the field. Were the annotation errors in this post found by a model? No. The 370 degenerate boxes and 23 out-of-bounds stem points were found with a pure geometric check against bounding_box, stem_x, and stem_y — fields that already exist on every Detection. No inference, no predictions, no model of any kind was involved. Is image_id a reliable way to find duplicate images in CropAndWeed? No. image_id is a 4-digit counter that resets to 0000 at the start of every recording session, so its repetition across the dataset (hundreds of times for values like 0002 or 0004) reflects the filename convention, not duplicate content. (recording_set, session) combined with fiftyone.brain.compute_uniqueness is the right way to check for visual similarity — note that session alone also isn't globally unique, since both recording sets reuse the same session-ID strings. Can I use CropAndWeed commercially? No. It's released under a custom non-commercial research license — academic research, teaching, and personal experimentation only. Redistributing the dataset itself isn't permitted either; derivative works, such as trained model weights, are. How long does it take to verify numbers like these? The full script behind this post — schema summary, session/capture-density stats, class-imbalance counts, the geometric QA checks, and a real fiftyone.brain.compute_uniqueness run over all 8,034 images — finishes in under 35 seconds on a GPU machine. Can whole-image embedding search find a rare class hiding in a busy scene? Not reliably. Across CLIP, Qwen3-VL-Embedding-2B, and CRADIOv4 — image-seeded search, text-prompt search, and unsupervised outlier detection — zero of eight held-out Black nightshade samples were recovered. A whole-image embedding represents the scene, not a single small instance within it. Patch-level embedding search (embedding each detection crop individually) is the fix, and it comes with a real ceiling of its own: a handful of seed patches can't cover every visual mode of a rare species. Can you fix annotation errors in FiftyOne without writing code? Yes. The in-App annotation editor lets you drag a mislabeled stem point back inside its bounding box, resize a detection, add an instance mask with a single positive click, and edit primitives like moisture or lighting from a dropdown — all in the sample view, no notebook required.

The Robot Grabbed the Wrong Block !Synthetic KubriCount scene showing multiple visually similar objects, illustrating how a grounding model can lock onto the wrong one. Synthetic KubriCount scene showing multiple visually similar objects, illustrating how a grounding model can lock onto the wrong one. A synthetic benchmark built to break foundation models, and a four-stage Voxel51 workflow — curate, annotate, generate, evaluate — for catching the failure before you deploy it. You’re about to hand a robot a natural-language instruction: “pick the small bolt, not the big one.” Not “pick a bolt.” Not “pick the bolt.” The small one, specifically, is sitting next to a bin full of the wrong one. That instruction only works if the model underneath it can ground it — lock onto the exact object you meant, not the object that happens to be biggest, or brightest, or most common in the bin. Robotics is currently making two big, correlated bets that this will just work out: that synthetic data can replace the teleop and real-world collection bottleneck, and that foundation models can do open-world perception well enough to follow instructions like that one. Both bets fail quietly. Synthetic pipelines ship artifacts nobody authored. Foundation models report a confident, wrong number. You usually find out after the GPU hours are already burned — or after the robot has grabbed the wrong block. A new benchmark called KubriCount — introduced in Count Anything at Any Granularity (Liu, Wu & Xie, SJTU) — turns that fuzzy fear into something you can measure. It reframes counting as a “verifiable prompt-following problem”: the question isn’t whether a model can output a plausible number; it’s whether it can count the correct set under an explicit, fine-grained prompt. The paper’s headline finding is blunt — models systematically “default to visually dominant or repetitive groups” when a prompt gets specific. That’s the robot grabbing the wrong block, expressed as a benchmark score. Counting is just the vehicle here. It’s the version of open-world grounding that’s easiest to read at a glance and to plot. The real subject of this post is the workflow: how you use FiftyOne to walk a synthetic dataset and a foundation model’s predictions through one continuous loop — curate → annotate → generate → evaluate — and catch the failure at each stage instead of finding out about it on a leaderboard. Everything below runs against a 6,736-sample FiftyOne-ready subset of KubriCount, Voxel51/KubriCount on Hugging Face, with predictions from NVIDIA’s open-world grounding model LocateAnything-3B already loaded in. Every number quoted is pulled directly from that dataset — not estimated. Key Takeaways - KubriCount’s full benchmark is 110,507 synthetic images, ~7.3 million annotated objects, 157 categories across 16 super-categories, built by a fully automatic four-stage pipeline: 3D asset curation (ShapeNetCore-v2 + TRELLIS-generated assets, ~58K assets), Kubric+Blender rendering, Nano-Banana-Pro image editing for sim-to-real realism, and Gemini-3-Pro VLM filtering for quality control. - The dataset defines five explicit counting granularity levels (L1 identity → L5 concept), each pairing a target set with a controlled distractor set that differs by exactly one semantic factor — this is what makes “count the correct set” a verifiable, gradeable task instead of a vibe. - Every ground-truth claim in this post is verified against the live dataset, not assumed: len(sample.example_boxes.detections) == sample.count holds for all 6,736 samples, zero violations — the synthetic labels really are self-consistent. - NVIDIA LocateAnything-3B, evaluated via dataset.evaluate_detections() (COCO protocol, IoU 0.5) against example_boxes, scores precision 0.33 / recall 0.28 on the Level-5 test splits, with 18 of 154 test-split categories (including teddy_bear and saw) sitting at exactly 0.00 recall. - Filtering the evaluated view down to predicted_count ≈ negative_count (and far from the true count) surfaces 94 scenes where the model counted the distractor instead of the target it was prompted for — the exact “wrong block” failure mode, reproducible with a four-line ViewField query. - Coloring precomputed SigLIP embeddings (siglip_viz) by split makes the generalization gap visible without a single line of clustering code: train and testA overlap, testB (entirely novel categories) sits off on its own. What KubriCount actually is !KubriCount's four-stage automatic pipeline: 3D asset curation, Kubric and Blender rendering, Nano-Banana-Pro image editing, and Gemini-3-Pro quality filtering. KubriCount's four-stage automatic pipeline: 3D asset curation, Kubric and Blender rendering, Nano-Banana-Pro image editing, and Gemini-3-Pro quality filtering. Each scene in KubriCount is a 1024×1024 synthetic image. Every image is produced by a four-stage automatic pipeline, and it’s worth naming the stages once because they map directly onto the curate → annotate → generate → evaluate loop this post walks through: 1. 3D asset curation. ~58K 3D assets across 157 categories, sourced from ShapeNetCore-v2 (53K assets, 55 categories with clean taxonomic labels) and supplemented with TRELLIS-family controllable 3D generation (~5K more assets, 102 new categories) to fill in the long tail. ~5K HDRI environment maps (Poly Haven + Text2Light) provide lighting diversity. 1. Prototype synthesis. Kubric + PyBullet + Blender render multi-object scenes with exact instance-level metadata — RGB, instance masks, 2D/3D boxes, center points — all authored by the simulator, not a human annotator. Object counts run 1 → 250 per scene (capped by Kubric’s 256-instance ID limit). 1. Consistent image editing. Nano-Banana-Pro refines textures and harmonizes lighting to narrow the sim-to-real gap, under a hard constraint: no instance may be added, removed, merged, or split. Whatever the simulator counted remains in the frame after editing. 1. Automatic quality filtering. Gemini-3-Pro inspects each edited image against the original prototype and masks, issuing a PASS/FAIL verdict. A single pass rejects ~20% of edits; three rounds of iterative re-editing bring the final rejection rate down to ~5%. Good to know. What are the five granularity levels? KubriCount’s core idea is to make “what counts as the target” explicit instead of implicit. Each level pairs a target set with an optional distractor set that differs by exactly one semantic factor: | Level | Granularity | Prompt example | Distractor | | --- | --- | --- | --- | | L1 | Identity | “Count all the dogs.” | None | | L2 | Attribute (size/color) | “Count the large cherries.” | Small cherries | | L3 | Category | “Count the cans.” | Bags | | L4 | Instance type | “Count backpack A.” | Backpack B | | L5 | Concept | “Count the lobsters.” | Octopuses | KubriCount's five counting granularity levels. Each pairs a target prompt with a distractor set that differs by exactly one semantic factor. Keep it honest. The full KubriCount benchmark spans all five levels across train (99,639 scenes), testA (5,462 scenes, unseen assets from seen categories), and testB (5,406 scenes, entirely unseen categories) — roughly 110K images and 198K counting queries in total. The 6,736-sample subset this post uses is narrower by design: train here is entirely Level 1 (“count everything,” no distractor), and both testA (924 samples) and testB (812 samples) are entirely Level 5 (the hardest disambiguation level). That’s the deliberate easy-to-hardest arc the rest of this post walks — it is not a representative sample of all five levels, and any number below that says “test” means “Level 5 test,” not “the full evaluation protocol from the paper.” Get the subset and open it: pip install -U fiftyone import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/KubriCount", persistent=True) session = fo.launch_app(dataset) The field names you’ll see throughout — verified against the live dataset’s own schema, not guessed: | Field | Type | What it holds | | --- | --- | --- | | split | StringField | train, testA (novel assets, seen categories), or testB (entirely novel categories) | | level | IntField | Granularity level, 1–5 (this subset: 1 and 5 only) | | category | StringField | The target prompt — what the model was asked to count | | count | IntField | Ground-truth number of target objects | | target_points | Keypoints | One center point per target object, len == count by construction | | example_boxes | Detections | Ground-truth target bounding boxes, one per target instance | | segmentation | Segmentation | Per-instance Kubric-rendered mask | | negative_category | StringField | The distractor prompt (empty for L1) | | negative_count | IntField | Ground-truth distractor object count | | negative_example_boxes | Detections | Ground-truth distractor bounding boxes | | locate_anything_pred | Detections | Predictions from NVIDIA LocateAnything-3B, prompted with category | | siglip_viz | Embedding coords | 2D UMAP projection of a SigLIP image embedding (brain key siglip_viz) | | predicted_count | IntField | len(locate_anything_pred.detections) | | abs_error | FloatField | \|predicted_count - count\| | Fields in the FiftyOne-ready KubriCount subset, verified against the live dataset's own schema. Two of those are load-bearing distinctions to keep straight for the rest of this post: example_boxes is ground truth, authored by the simulator. locate_anything_pred is model output — everything that follows about precision, recall, and “the model got it wrong” refers to that field, never to the labels themselves. Step 1: Curate — open it and look before you burn compute The first move with any new dataset, synthetic or real, is to open it and look — not read a README, not trust a data card. FiftyOne’s App gives you three views into the data’s shape with zero configuration. !FiftyOne App Histograms and Embeddings panels showing KubriCount count, category, and level distributions during dataset curation. FiftyOne App Histograms and Embeddings panels showing KubriCount count, category, and level distributions during dataset curation. The Histograms panel auto-plots the distribution of whatever field you pick and updates live to the current view — no dashboard code required to get the shape of the data: - count: object density per scene runs from 1 to 248 — these are dense, cluttered scenes by design. - category: 157 distinct target categories in this subset. - level / split: this is the structural fact most people skim past. train (5,000 samples) is entirely Level 1 — “count everything, no distractor.” Both testA (924) and testB (812) are entirely Level 5 — the hardest disambiguation level — with two categories, each containing multiple instance types. The data itself ramps from the easiest possible task to the hardest. For the visual structure, color the precomputed embeddings — no clustering code needed: import fiftyone.brain as fob # Already computed on the Hub subset; shown here for reproducing on your own data. fob.compute_visualization( dataset, embeddings="siglip_embeddings", method="umap", brain_key="siglip_viz", ) Color the Embeddings panel by category, and the 157 classes are separated by visual appearance. Color it by level, and the L1/L5 split falls out as two visually distinct regions. On a real synthetic-data pipeline, this is the exact panel where you’d notice a category your generator over- or under-produced — before you’ve spent a single GPU-hour training on it. Step 2: Annotate — trust no label, verify it Synthetic labels have a reputation for being “free and perfect” — the simulator knows exactly what it rendered, so why would you check? Because “supposed to be perfect” and “verified perfect” are different claims, only one of them is defensible in a paper or a design review. KubriCount’s design gives you a free invariant to check: every target object gets exactly one ground-truth box, so len(example_boxes.detections) must equal count for every sample, with no exceptions. from fiftyone import ViewField as F violations = dataset.match( F("example_boxes.detections").length() != F("count") ) print(violations.count(), "/", dataset.count()) 0 / 6736 Zero violations across all 6,736 samples. That’s not an assumption carried over from a data card — it’s a five-second query against the live dataset. Save it as a view so it’s one click away, not a query you retype every time: dataset.save_view( "integrity_box_count_eq_count", violations, description="len(example_boxes) != count — should always be empty", ) !FiftyOne App toggling between ground-truth target boxes and distractor boxes on a KubriCount scene to verify synthetic labels. FiftyOne App toggling between ground-truth target boxes and distractor boxes on a KubriCount scene to verify synthetic labels. That distinction is what matters for robotics data: example_boxes is the target you were prompted for, negative_example_boxes is the look-alike you have to ignore. On real robot data, this exact pattern — verify an invariant, then eyeball the ambiguous cases — is how you’d hunt the bad 2% of a labeling run and route the genuinely unclear scenes to a human annotator instead of shipping them straight into training. Step 3: Generate — find the gap you’d be generating blind into Synthetic generation is how robotics gets around the teleop and real-world collection bottleneck. But generating more data without considering what you already have just amplifies any baked-in bias. Before you generate the next batch, look at the distribution shift you already have on hand. KubriCount’s test splits are built for exactly this: testA holds out unseen assets from categories the model has already seen in train; testB holds out entire categories the model has never seen at all. Recolor the same embeddings panel by split: session.view = dataset # full dataset, then recolor Embeddings panel by `split` !SigLIP embeddings in FiftyOne colored by split, with train and testA overlapping and testB set apart, revealing the generalization gap. SigLIP embeddings in FiftyOne colored by split, with train and testA overlapping and testB set apart, revealing the generalization gap. train and testA overlap — same categories, new object instances, no real surprise for the model. testB sits off on its own, visually distinct. That separation is the generalization gap, made visible by a single recolor with no extra dimensionality-reduction code. For a robotics dataset, this is exactly how you’d sanity-check whether your “held-out test set” is actually testing generalization or just re-sampling the training distribution with new random seeds. Make it concrete with similarity search instead of a 2D scatterplot alone: testB = dataset.match(F("split") == "testB") dataset.save_view("testB_novel_categories", testB, description="Entirely novel categories") # In the App: right-click a testB sample -> Sort by similarity # (reuses the same siglip embeddings, no extra brain run needed) !FiftyOne similarity search sorting a testB KubriCount sample by SigLIP embedding to surface nearest neighbors among novel categories. FiftyOne similarity search sorting a testB KubriCount sample by SigLIP embedding to surface nearest neighbors among novel categories. That’s the honest version of “how far are you asking the model to extrapolate”: not a number, but the actual nearest neighbor sitting a visible distance away. Step 4: Evaluate — aggregate metrics lie, so look at the boxes This is where the foundation-model bet gets tested. The model under test is NVIDIA LocateAnything-3B, an off-the-shelf open-world grounding detector, applied to every sample with category as the text prompt: import fiftyone as fo import fiftyone.zoo as foz foz.register_zoo_model_source( "https://github.com/Burhan-Q/fiftyone-locate-anything", overwrite=True, ) foz.download_zoo_model( "https://github.com/Burhan-Q/fiftyone-locate-anything", model_name="nvidia/LocateAnything-3B", ) model = foz.load_zoo_model( "nvidia/LocateAnything-3B", operation="grounding", single_instance=False, generation_mode="fast", max_new_tokens=8192, ) dataset.apply_model( model, prompt_field="category", label_field="locate_anything_pred", batch_size=64, num_workers=16, skip_failures=False, ) Summed across the whole subset, the model returned 233,729 predicted boxes against 334,983 ground-truth boxes — undercounting on average, but that single aggregate hides everything interesting. Run a proper COCO-style detection evaluation against the target boxes instead of eyeballing totals: results = dataset.evaluate_detections( "locate_anything_pred", gt_field="example_boxes", eval_key="eval_boxes", method="coco", iou=0.5, classwise=True, ) results.print_report() Restricted to the Level-5 test splits (testA + testB, 1,736 samples — where the target/distractor disambiguation actually gets hard), that comes out to: precision: 0.33 recall: 0.28 The model finds fewer than a third of the objects it’s supposed to find, and two-thirds of what it does return is wrong. Open the per-class breakdown and it gets worse: 18 of 154 test-split categories sit at exactly 0.00 recall — teddy_bear and saw among them. It’s not that the model is bad at counting in general (Level 1, “count everything,” is comparatively fine). It’s bad at counting the thing you specifically asked for the moment a look-alike distractor is in frame. !FiftyOne view of a KubriCount scene where LocateAnything-3B returned 34 distractor vessels instead of the 6 target boats it was prompted for. FiftyOne view of a KubriCount scene where LocateAnything-3B returned 34 distractor vessels instead of the 6 target boats it was prompted for. The model counted the distractor Aggregate precision/recall still doesn’t show you why it fails. The specific failure the paper predicts — models defaulting to the visually dominant or repetitive group — has a signature you can query directly: scenes where predicted_count lands close to negative_count (the distractor’s true count) but far from count (the target’s true count). counted_the_distractor = dataset.match( F("split").is_in(["testA", "testB"]) & (F("negative_count") > 0) & ((F("predicted_count") - F("negative_count")).abs() <= 2) & ((F("predicted_count") - F("count")).abs() > 3) ) print(counted_the_distractor.count()) # 94 dataset.save_view( "counted_the_distractor", counted_the_distractor, description="Model returned the distractor's count, not the target's", ) 94 94 scenes where the model grounded onto the wrong object entirely. Open one and it’s not subtle. One sample prompts category = "boat" — 6 boats are actually in the scene — while 33 vessel objects (the distractor class) sit in the same frame. The model returned 34 boxes. It didn’t miscount the boats; it counted the vessels and reported that number back as if the prompt had been “count the vessels.” That’s the classic “pick the target, not the look-alike” instruction failing on a current open-world model — and the only reason it’s diagnosable at all is that someone filtered to the actual scenes instead of reading a leaderboard row. A second, related failure mode is worth a look too: scenes where the model counted target and distractor together, with no disambiguation at all. The query below finds 49 of them: grabbed_everything = dataset.match( F("split").is_in(["testA", "testB"]) & ((F("predicted_count") - (F("count") + F("negative_count"))).abs() <= 2) & ((F("predicted_count") - F("count")).abs() > 2) ) print(grabbed_everything.count()) # 49 49 !FiftyOne view of a KubriCount scene where LocateAnything-3B counted target and distractor objects together, showing the no-disambiguation failure mode. FiftyOne view of a KubriCount scene where LocateAnything-3B counted target and distractor objects together, showing the no-disambiguation failure mode. The loop doesn’t end at evaluation Four stages, but really one loop: curate, annotate, generate, evaluate, and back to curate again with what the evaluation just told you. The 94 scenes in counted_the_distractor aren’t only a finding. They’re a work order. They tell you exactly which category pairs (boat/vessel, pillow/clock, spoon/candle) need more or harder synthetic examples next, which labels need a second look, and which embedding regions to inspect for more of the same failure. FiftyOne doesn’t collect your data, train your policy, or close your robot’s control loop. It’s the layer in between — for synthetic data, for real data, and for foundation-model failures alike, whether the failure mode is “counted the wrong thing” or “picked the wrong block.” You can’t fix what you can’t see. So look first. Try it yourself - Dataset (this subset): Voxel51/KubriCount on Hugging Face - Full benchmark + generation pipeline: Verg-Avesta/KubriCount on GitHub - Paper: Count Anything at Any Granularity, Liu, Wu & Xie (2026) - Project page: verg-avesta.github.io - Model used for predictions: NVIDIA LocateAnything-3B via Burhan-Q/fiftyone-locate-anything - FiftyOne: pip install -U fiftyone · docs Frequently Asked Questions Is example_boxes the same as the exemplar crops described in the original KubriCount release? Not in this subset. The general KubriCount project describes example_boxes upstream as a small set of few-shot exemplar crops (2–8 per scene). In this FiftyOne-ready subset, example_boxes has been verified directly against the live data to be exhaustive, one-per-instance ground truth for all 6,736 samples) — that’s what makes it usable as detection-evaluation ground truth in the steps above. If you’re working from a different KubriCount release, check the field’s description in the FiftyOne App sidebar before assuming either behavior. Why does train only have Level 1 and the test splits only have Level 5? That’s a deliberate choice for this particular subset, not a property of the full benchmark. The full KubriCount spans all five levels in every split (~100K train images, ~5K each in testA/testB). This subset selects the two extremes — the easiest task for training exposure and the hardest task for evaluation — to make the easy-to-hard arc legible in a small sample. If you need the middle levels (L2–L4), pull the full dataset from liuchang666/KubriCount. Are the precision/recall numbers (0.33 / 0.28) from the paper? No — those are computed in this post via dataset.evaluate_detections() (COCO protocol, IoU 0.5) comparing NVIDIA LocateAnything-3B’s locate_anything_pred field against the example_boxes ground truth, on this 1,736-sample Level-5 test slice. The paper’s own benchmark table evaluates a different, larger set of MLLMs and specialist counting models (MAE/RMSE protocol) on the full benchmark and does not include LocateAnything-3B. Don’t cite 0.33/0.28 as a paper result — it’s this post’s own model-in-FiftyOne evaluation. What does abs_error mean and how is it different from eval_boxes_fp / eval_boxes_fn? abs_error is a scalar, |predicted_count - count| — how far off the total object count was, with no notion of which boxes were right. eval_boxes_tp/eval_boxes_fp/eval_boxes_fn come from the box-level COCO evaluation and tell you whether the individual predicted boxes actually matched ground-truth boxes (by IoU) — which is what lets you compute precision/recall and catch the “right count, wrong objects” failure mode that a bare count-error metric would miss entirely. Does this scale past a 6.7K-sample subset? Yes — the App’s grid and panels paginate server-side, so millions of samples behave the same way interactively. This post uses the 6,736-sample subset specifically because it’s the one with precomputed embeddings, predictions, and evaluation results ready to go; the full benchmark is ~110,507 images and ~7.3 million annotated objects.