GoogleのEmbeddingGemma 2をOpenCLIP風に使えるPythonライブラリを作ってPyPIに公開してみた
はじめに
GoogleのEmbeddingGemma 2を、OpenCLIPに近いインターフェースで扱えるPythonライブラリ「openclip-gemma」を開発し、PyPIに公開しました。
GitHub: kotadymo/openclip-gemma
PyPI: pypi.org
pip install openclip-gemma
openclip-gemmaは、EmbeddingGemma 2を使った画像・テキストの埋め込み生成を、create_model_and_transforms()、get_tokenizer()、encode_image()、encode_text()といった、OpenCLIPでおなじみのAPIで実行できるようにするライブラリです。
画像とテキストの類似度計算、画像検索、Zero-shot分類などの処理を、比較的シンプルなPythonコードで試せることを目指しています。
今回は、開発した背景、ライブラリの特徴、インストール方法、使い方、現時点での制約について紹介します。
追記
本ライブラリの名称「openclip-gemma」は、OpenCLIP風のAPIでEmbeddingGemma 2を利用できるという特徴から付けたものです。
本ライブラリは個人開発の非公式プロジェクトであり、Google、Google DeepMind、OpenCLIPプロジェクトとは提携・協力関係になく、公式に承認されたものでもありません。
既存プロジェクトの名称や商標への配慮から、今後ライブラリ名を変更する可能性があります。変更する場合は、GitHubで移行方法をご案内します。
1. なぜ作ったのか
本音
この記事はほぼすべてAIに書かせましたが、こちらに作った理由の本音を記載しておきます。
個人製作の中でよくCLIPを使用しています。特定のシーンを検出するのにチューニングに苦労しているのですが、先々週このEmbeddingGemma 2というモデルが登場しました。いろいろと調べていると、CLIPと同様の画像とテキストの比較ができるということがわかりました。しかし、CLIPとは異なる考えのもと作成されているので、CLIPと同様に使えるわけではありません。
なので、「CLIPと同じような感覚でEmbeddingGemma 2を扱えるライブラリがあったら便利じゃね？」という安易な考えから、作ってみることにしました。ちなみに、今回の開発ではコードの実装をほぼすべてAIに任せています。私自身、Pythonライブラリを公開するのは初めてです。
OpenCLIPの使いやすいAPI
OpenCLIPは、画像とテキストを同じ埋め込み空間で扱うCLIP系モデルを利用するためのライブラリです。
例えば、モデルの初期化、画像の前処理、テキストのトークン化、埋め込み生成を次のような流れで実行できます。
import open_clip
model, _, preprocess = open_clip.create_model_and_transforms(
"ViT-B-32",
pretrained="..."
)
tokenizer = open_clip.get_tokenizer("ViT-B-32")
このインターフェースは直感的で、画像とテキストを組み合わせた処理を実装しやすいと感じています。
一方で、EmbeddingGemma 2はOpenCLIPのモデルとしてそのまま扱えるわけではありません。
そこで、EmbeddingGemma 2の公式な推論処理を維持しながら、OpenCLIPに近い書き方で利用できないかと考えました。
EmbeddingGemma 2を利用したい
EmbeddingGemma 2は、Googleの画像・テキスト対応の埋め込みモデルです。
画像とテキストの両方を768次元の特徴ベクトルに変換でき、ベクトル間の類似度を利用することで、画像と文章の関連性を計算できます。
例えば、次のような用途を想定できます。
- テキストから関連する画像を検索する
- 画像に近い説明文を選択する
- 複数の文章候補と画像の類似度を比較する
- 画像とテキストを利用した検索システムを試作する
そこで、OpenCLIPに似たAPIでEmbeddingGemma 2を扱える、推論専用のラッパーライブラリを作成しました。
2. openclip-gemmaの特徴
主な特徴は以下のとおりです。
| 項目 | 内容 |
|---|---|
| 対応モデル | Google EmbeddingGemma 2 |
| 画像入力 | 対応 |
| テキスト入力 | 対応 |
| 日本語テキスト | 対応 |
| 埋め込み次元 | 768次元 |
| API | OpenCLIP風 |
| バッチ推論 | 対応 |
| Zero-shot類似度計算 | 対応 |
| 推論エンジン | PyTorch |
| 学習・Fine-tuning | 非対応 |
| 動画・音声 | 本ライブラリでは非対応（モデル自体は対応） |
公式モデルの画像前処理、トークナイザー、プロンプト、プーリング、正規化の処理を維持するように実装しています。
また、モデル重みはパッケージに同梱していません。実行時にHugging Faceから取得します。
モデルには特定のリビジョンを指定しており、モデル側の更新によって意図せず推論結果が変化するリスクを抑えています。
なお、OpenCLIPの完全な互換実装ではありません。 OpenCLIPのすべてのモデルやAPIを実行できるわけではなく、EmbeddingGemma 2専用の推論ラッパーです。
3. インストール
現時点で検証済みの環境は以下です。
- Windows 11 x64
- Python 3.12
- CPU（FP32およびBF16）
Linux、macOS、CUDAについては、v0.1.0時点では動作検証が完了していません。
まず仮想環境を作成します。
python -m venv .venv
Windowsの場合は以下で有効化できます。
.\.venv\Scripts\Activate.ps1
続いてインストールします。
python -m pip install openclip-gemma==0.1.0
PyTorchなど、推論に必要な依存ライブラリもインストールされます。
初回のモデル読み込みでは、Hugging Faceからモデルファイルを取得するため、ネットワーク接続とダウンロード時間、ディスク容量が必要です。
4. 基本的な使い方
4.1 モデルの読み込み
まず、EmbeddingGemma 2を読み込みます。
import torch
import openclip_gemma
device = "cpu"
model, _, preprocess = (
openclip_gemma.create_model_and_transforms(
"embeddinggemma-2",
device=device,
)
)
model.eval()
create_model_and_transforms()は、OpenCLIPに近い形式でモデルと画像前処理関数を返します。
今回はCPUを明示しています。
4.2 画像を768次元のベクトルに変換する
Pillowで画像を読み込み、画像の埋め込みを生成します。
from PIL import Image
image = Image.open("sample.jpg").convert("RGB")
image_tensor = (
preprocess(image)
.unsqueeze(0)
.to(device)
)
with torch.inference_mode():
image_features = model.encode_image(
image_tensor,
normalize=True,
)
print(image_features.shape)
出力例：
torch.Size([1, 768])
1枚の画像から768次元の特徴ベクトルが得られます。
ここで注意したいのは、画像に対してライブラリが返すpreprocessを使うことです。
EmbeddingGemma 2固有の画像前処理を想定しているため、別のCLIPモデル用に正規化したテンソルをそのまま渡すことはできません。
4.3 日本語テキストをベクトルに変換する
続いて、テキストの埋め込みを生成します。
tokenizer = openclip_gemma.get_tokenizer(
"embeddinggemma-2"
)
texts = [
"机の上に置かれたノートパソコン",
"公園を走る犬",
"青空の下にある建物",
]
tokens = tokenizer(texts).to(device)
with torch.inference_mode():
text_features = model.encode_text(
tokens,
normalize=True,
)
print(text_features.shape)
出力例：
torch.Size([3, 768])
3つの日本語テキストそれぞれについて、768次元の特徴ベクトルが得られます。
get_tokenizer()を利用することで、モデルが必要とするトークン化を実行できます。
5. 画像と文章の類似度を計算する
ここが今回のライブラリで特に試していただきたい部分です。
画像と複数の説明文を比較し、どの文章が画像に近いかを計算します。
import torch
from PIL import Image
import openclip_gemma
device = "cpu"
# モデルの読み込み
model, _, preprocess = (
openclip_gemma.create_model_and_transforms(
"embeddinggemma-2",
device=device,
)
)
model.eval()
# 画像の読み込み
image = Image.open("sample.jpg").convert("RGB")
image_tensor = preprocess(image).unsqueeze(0).to(device)
# 比較したいテキスト
candidate_texts = [
"机の上に置かれたノートパソコン",
"公園を走る犬",
"青空の下にある建物",
]
# 分類タスク用の公式プロンプトを使用
tokenizer = openclip_gemma.get_tokenizer(
"embeddinggemma-2",
prompt_name="Classification",
)
text_tokens = tokenizer(candidate_texts).to(device)
# 埋め込み生成と類似度計算
with torch.inference_mode():
image_features = model.encode_image(
image_tensor,
normalize=True,
)
text_features = model.encode_text(
text_tokens,
normalize=True,
)
similarities = image_features @ text_features.T
# 結果表示
scores = similarities[0].tolist()
for text, score in zip(candidate_texts, scores):
print(f"{text}: {score:.4f}")
best_index = similarities[0].argmax().item()
print("最も類似度が高い文章:")
print(candidate_texts[best_index])
image_features @ text_features.Tは、正規化された画像・テキストの特徴ベクトル間の内積を計算しています。
これはコサイン類似度として利用できます。
類似度が高いほど、その画像と文章がモデルの埋め込み空間上で近いことを意味します。
ただし、類似度は分類確率ではありません。
例えばスコアが0.75であっても、「75%の確率で正解」という意味ではありません。
また、最も高いスコアの文章が常に正しいとは限りません。用途ごとに候補文の設計や精度評価が必要です。
6. OpenCLIPからの移行イメージ
既存のOpenCLIPを利用した推論コードでは、次のような書き方をしていることがあります。
import open_clip
model, _, preprocess = (
open_clip.create_model_and_transforms(
"ViT-B-32",
pretrained="...",
)
)
tokenizer = open_clip.get_tokenizer("ViT-B-32")
openclip-gemmaの場合は次のようになります。
import openclip_gemma as open_clip
model, _, preprocess = (
open_clip.create_model_and_transforms(
"embeddinggemma-2",
device="cpu",
)
)
tokenizer = open_clip.get_tokenizer(
"embeddinggemma-2"
)
初期化部分の構造が似ていることが分かると思います。
さらに、次のAPIも利用できます。
model.encode_image(...)
model.encode_text(...)
このため、画像とテキストを別々にエンコードして類似度を計算するような処理を、比較的近い構成で記述できます。
ただし、モデルごとの前処理、トークン化、プロンプト、埋め込み特性は異なります。
既存のOpenCLIPコードをimport文だけ変更すれば必ず動くというものではありません。
詳細は互換性レポートに整理しています。
7. 実装・精度検証について
今回の開発では、APIを似せるだけではなく、公式の推論結果と大きく乖離しないことを重要視しました。
そのため、公式のSentence Transformers経由の推論結果と、openclip-gemmaの出力を比較しています。
Windows 11、Python 3.12、CPU FP32で測定した最大絶対誤差は、以下のとおりでした。
| 比較対象 | 最大絶対誤差 |
|---|---:|
| 画像埋め込み | 約2.74 × 10⁻⁷ |
| テキスト埋め込み | 約2.81 × 10⁻⁶ |
| 画像・テキスト類似度 | 約1.19 × 10⁻⁶ |
特定のモデルリビジョンとテスト入力・実行環境における比較結果ですが、公式の推論経路と非常に近い値が得られています。
また、リリース前には以下を確認しています。
- pytest：75件成功、9件スキップ
- 実モデルを使ったCPU統合テスト：7件成功、CUDA専用の2件をスキップ
- Ruffによる静的チェック
- wheel・sdistのビルドと内容検査
- TestPyPIからクリーン環境へのインストール
- 画像・日本語テキストを利用した実モデル推論
- GitHub ActionsによるCIとPyPIへの自動公開
ただし、これらは推論処理の整合性や配布の正常動作を確認するものであり、個別の画像分類・検索タスクにおける精度を保証するものではありません。
8. 現時点での制約
v0.1.0は初期リリースのため、機能を限定しています。
特に次の点には注意してください。
対応モデルはEmbeddingGemma 2のみです。
任意のCLIPモデルを読み込むことはできません。
推論専用です。
学習やFine-tuningには対応していません。
動画・音声には対応していません。
現在は静止画像とテキストの埋め込み生成を対象としています。
CUDA、Linux、macOSは未検証です。
CUDA向けのコードは実装していますが、GPU実機での動作確認は今後の課題です。
OpenCLIPの完全互換ではありません。
forward()、get_logits()など、一部のOpenCLIPで使われる機能は提供していません。また、OpenCLIPのlogit_scaleを再現するものではありません。
9. 今後について
今後は、実際の利用状況を見ながら、対応環境の拡大や使い勝手の改善を検討したいと考えています。
例えば、CUDA環境での実機検証、Linux・macOSでの動作確認、サンプルコードやドキュメントの充実などが考えられます。
初期バージョンのため、不具合や改善点があればGitHub Issuesで報告いただけるとうれしいです。
github.com/kotadymo/openclip-gemma/issues
おわりに
今回は、EmbeddingGemma 2をOpenCLIP風のAPIで扱えるPythonライブラリ「openclip-gemma」を紹介しました。
画像とテキストの埋め込み生成や類似度計算を、できるだけシンプルなコードで実行できるようにしたライブラリです。
EmbeddingGemma 2を試してみたい方や、OpenCLIPに近い書き方で画像・テキスト埋め込みを扱いたい方の参考になれば幸いです。
まだv0.1.0の初期段階ですが、ぜひ試してみてください。
GitHub: kotadymo/openclip-gemma
PyPI: pypi.org
なお、本ライブラリは個人開発の非公式プロジェクトであり、Google、Google DeepMind、OpenCLIPの公式実装・公式配布物ではありません。ライブラリ本体はApache-2.0で公開しています。モデル重みは同梱しておらず、利用時にはGoogleのモデルに適用される別途の利用条件をご確認ください。
Side Quests Grow on Trees (Leave Them Where They Grow) 🌿
---
description: A kids' plant treasure hunt for Android with on-device BioCLIP and TinyCLIP, Gemma 4 hints built locally, and 0 of 267 toxic test photos passing as a match.
---
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
TL;DR
- Wild Find is a native Android treasure hunt for kids 8 and up, built for (and tested by) my family, that sends kids after real plants common to their area.
- Two open-weight models run on the phone, BioCLIP 2.5 Mobile and TinyCLIP, and no images ever leave the device.
- Gemma 4 26B runs locally in the build pipeline to precompile hints and hazard warnings, and every win still comes down to deterministic code: 0 of 267 toxic test photos passed as a match.
- 🎵 I love assigning a theme song to my builds, just because it's fun. This one is Pinecones by Bug Hunter, and it's fabulous!
- 🎬 Watch the demo · 💻 Browse the repo · 📱 APK releases
What I Built
The idea for this game came from a family powwow, where every idea built off the last until I was satisfied that we had outlined something I could realistically build.
Wild Find is a treasure hunt game designed for kids 8 years and up to play with their parents. Even though my son is much older than that, I still had a lot of fun building and testing!
!Screenshot Wild Find home screen.
This is not a quick game. It's designed to keep kids, both young and old, outside looking for plants as long as they can stand it.
The app gives you three randomly selected plants for a region that is set by an adult user based on location or picked from a map.
The kid is then tasked to find one of three plants common to their specific area of the world. When they think they've found one, they capture a picture without picking the plant, or they can skip an extra hard one. The game is finished when you find all three plants from your list.
!Wild Find's hunt list for Georgia in October: blue mistflower, eastern redbud, and American beautyberry, each with a short description.
My son and sister both played the game without any instruction from me. We all learned that the previously deferred in-game hints were a necessity to prevent users from getting frustrated by the game. Also, just because my region contains those plants does not mean that my yard does. So I added in skip functionality for each plant to keep it kid-friendly.
If the app flags a potentially hazardous plant, it warns the user not to touch it — which is a general safety rule in the game, anyway. There are 85 total hazards: seven in North America that are always on, like poison ivy, oak, and sumac, and another 78 that are only on when iNaturalist says they're nearby.
!Wild Find's hazard warning over a poison ivy photo: That might be a plant we leave extra space around. Its oil causes an itchy and painful rash. Step back and look somewhere else.
I'm most proud of the latency numbers I was able to accomplish with two open-weight models: BioCLIP 2.5 Mobile, a plant-focused model distilled from the larger BioCLIP 2.5, and TinyCLIP, a tiny general-purpose model that checks whether there's a plant in the frame at all. Together, they take a median of 189 ms per frame on my S24 Ultra.
As always, I built this app to be accessible from the start: big touch targets, labels for TalkBack, no color-only signals, animations that follow the reduced-motion settings, and hints and warnings that are still there when text is set to 200% size.
The app rounds the user's rough coordinates to the nearest whole degree before sending them to iNaturalist, which returns a list of regional plants, each with a short description of its type and up to three hints prebuilt from Wikipedia and USDA PLANTS. No photos or saved gameplay progress leave the device. iNaturalist also sees the request's IP address, as disclosed on the grown-ups page.
Demo
{% embed youtu.be %}
🎵 Theme song: Pinecones by Bug Hunter
> 📱 Want to play it yourself? Grab the Android APK (Android 11+) from the v1.0.0 release.
Code
{% embed anchildress1/wild-find %}
> ⚖️ This project is licensed under MIT.
I build with AI, including Claude, Codex, and Copilot — which ran out of credits before I had finished the build. There are plenty of distinct design decisions you can see in the code throughout.
First, I don't want a toxic or hazardous plant to win a hunt. I guard against this with a "blocker," where every local toxic or hazard species is checked in the photo capture. The best match must beat them all by 0.034 when the full frame is recognized as a plant, or 0.048 when only the circle is recognized. A blocker also scores on the circle alone, so a toxic plant that only fills the circle can't hide behind the rest of the photo. I tested this until 0 of 267 toxic plant photos passed, while real finds went from 97 to 108 of 198.
override fun score(reticle: FloatArray, full: FloatArray?): GoalScore {
fun s(row: Int) = if (full == null) {
table.dot(row, reticle)
} else {
(table.dot(row, reticle) + table.dot(row, full)) / 2
}
val margin = if (full == null) MARGIN else FULL_MARGIN
val scores = eligible.map(::s)
val best = scores.max()
val top = eligible[scores.indexOf(best)]
// A toxic plant that fills only the circle must not be diluted by the full frame, so a blocker scores the
// larger of its mean and its reticle cosine.
fun guard(row: Int) = if (full == null) s(row) else maxOf(s(row), table.dot(row, reticle))
val blocker = blockers.maxOfOrNull(::guard) ?: Double.NEGATIVE_INFINITY
val score = s(target)
// A tie for top-1 is no pass, whichever rows tie.
val met = scores.count { it == best } == 1 && genus[top] == genus[target] && best - blocker >= margin
return GoalScore(met, score, 1 + scores.count { it > score })
}
/**
* The reticle-only margin, for a frame whose full frame isn't a plant: the smallest round margin that let 0 of
* 180 local toxic photos pass on Oct 7 (a poison ivy photo read as beautyberry won by 0.0477).
*/
const val MARGIN = 0.048
/**
* The margin on the reticle and full-frame mean (day-5 rule R1h): the 0.024 tune boundary plus 0.01 headroom.
* With blockers floored at their reticle cosine it lets 0 of 267 toxic photos pass, with the closest toxic
* photo 0.020 under the margin, and raises target passes from 97 to 108 of 198 (`blocker_guard.log`).
*/
const val FULL_MARGIN = 0.034
Second, according to BioCLIP's model card, it matches an image against a fixed table of known species, and it's strongest at the genus level. The same card says "Do not use this for toxicity or edibility decisions," which is why the toxicity flags come from Wikipedia and USDA text, not the model.
So the app takes the full local list, asks BioCLIP to rank the capture image against each local plant species, and then deterministic checks run to determine if a match was made or not.
The code uses BioCLIP's scores for each plant, averaged across the circle and the whole photo when TinyCLIP sees a plant in the whole photo (for example, how much does this look like water oak?), to determine the top species and whether that top species is in the target's genus (because this is only the easy level; future iterations would increase the difficulty).
The 0.034 margin originally came from an earlier rule's 0.024 tuning boundary plus 0.01 of headroom. After adding the circle-only blocker guard, I tested the rule again. Lowering the margin to 0.011 allowed two toxic photos through in the fresh test set, so I kept 0.034.
That costs some legitimate finds, but the closest toxic test photo now falls 0.020 below the margin instead of just 0.0028.
A single matching frame still isn't a win, either. The app needs three matching frames in a row before it counts a find.
The Day 5 results have every rule I tried and every photo.
fun of(sightings: List<Sighting>): LocalList {
val floor = max(MIN_SIGHTINGS.toDouble(), SHARE * sightings.sumOf { it.count })
val byRow = sightings.mapNotNull { s ->
rowOf(s.scientific)?.let { it to s }
}.groupBy({ it.first }, { it.second })
val counted = byRow.mapValues { (_, names) -> names.sumOf { it.count } }.filterValues { it >= floor }
val eligible = counted.mapNotNull { (row, count) ->
val common = byRow.getValue(row).sortedByDescending { it.count }
.firstNotNullOfOrNull { it.common?.trim()?.takeIf(::isKidName) }
if (table[row].playable && common != null) Eligible(row, common, count) else null
}.sortedWith(compareByDescending<Eligible> { it.count }.thenBy { it.row })
// Any sighting blocks: the floor-only blockers let 87 of 180 toxic photos pass as some target on Oct 7.
val blockers = byRow.keys.filterNot { table[it].playable }.sorted().toIntArray()
// A hunt takes one target per genus, so three species of two genera still can't fill it; a row held back from
// picks can't fill a slot either.
val genera = eligible.filter { table[it.row].target }.distinctBy { table[it.row].genus }.size
return LocalList(eligible, blockers, needsWiden = genera < MIN_TARGETS)
}
Also, there were four plants BioCLIP couldn't reliably recognize in testing: sweetgum, tulip tree, persimmon, and winged sumac. So I pulled them all out of the hunt. They will still count when the app compares plants, though, so nothing sneaks through as one of those.
There are plenty of other examples in the repo, too. Check out these pages, if you're curious.
| File | What it does | Why it's code, not AI |
|---|---|---|
| Verify.kt | The verdict table: every message a kid sees after a capture, first matching row wins | One readable rule table decides the star, with no model judgment |
| HazardCheck.kt | Warns on 7 fixed hazards, including poison ivy, oak, and sumac, plus 78 regional ones, using local iNat sightings | The warning is a fixed cutoff that was tested on 52 hazard photos, not a model's opinion |
| toxicity.py | Flags toxic plants from Wikipedia sentences and USDA ratings | CLIP models scored toxicity at coin-flip odds (AUC 0.42–0.65), so text matching does the job |
| assets.py | Refuses to build if any plant is missing its toxicity flag, and never gives a toxic or hazard plant a hint | The build fails closed: a safety gap stops the APK, it doesn't ship it |
| RegionKey.kt | Rounds the location to a whole degree before anything leaves the phone | Privacy for kids comes from a single rounding function |
I tested the hazard warning separately, too. It caught 48 of 52 hazard photos and only warned on 3 of 253 regular plants by mistake. Worldwide, using a fresh set of images, it caught 41 of 52, which isn't perfect. And it's exactly why the game never tells a kid that a plant is safe to touch (hazard results).
Also, this build includes 314 Kotlin unit tests, 350 Python tests, and another 120 tests that run on device (67 device and 53 end-to-end). The automated checks pass, and it survived several rounds of manual testing on different devices, as proven by my son playing barefoot at the edge of our property.
!My son playing Wild Find barefoot at the edge of our property
How I Built It
The app is separated into two distinct sections: the build pipeline and the app itself that runs on device.
| Open-source AI | Type | Runs on | Job |
|---|---|---|---|
| BioCLIP 2.5 Mobile | Open-weight model | Phone | Scores each capture against the local plant list |
| TinyCLIP ViT-8M | Open-weight model (MIT) | Phone | Checks there's a plant in the frame at all |
| ONNX Runtime | Open-source inference runtime | Phone | Runs both models offline, with its telemetry switched off |
| BioCLIP 2.5 (ViT-H) + OpenCLIP | Open-weight model + framework | Laptop, build time | Writes the tutorial vectors and the hazard rows BioCLIP's table is missing |
| Gemma 4 26B | Open-weight model | Laptop, build time | Writes the hints and hazard warnings |
| Ollama | Local inference | Laptop, build time | Serves Gemma with no cloud API calls |
I looked into training a new model based on BioCLIP that could identify non-plants or poisonous plants, and could return hints inside the game. Based on research and model documentation already published on Hugging Face (Gemma 4 E2B, BioCLIP 2.5 Mobile, BioCLIP 2), I decided that would probably waste a lot of time and still not get me the results that I wanted.
Early testing had already proved a larger model would slow down gameplay. Gemma 4 E2B is a 2.6 GB file next to BioCLIP Mobile's 47 MB, and it took 2.3 to 2.9 seconds per photo, while BioCLIP and TinyCLIP together take a median of 189 ms per frame.
Instead, I chose to precompile a list of 4,272 plants, including the 2,093 not flagged as toxic or hazardous, and build that list into the released APK. First the build pipeline runs Gemma 26B against plant Wikipedia articles to determine hints based on hint type.
| Hint type | Plant | Hint | Source |
|---|---|---|---|
| Place | Blue mistflower | "Look in sandy woodlands and clearings." | Wikipedia |
| Light | Water oak | "It does not like shade." | Wikipedia |
| Ground | Christmas fern | "It grows in moist habitats." | Wikipedia |
| Nearby | American sweetgum | "It grows near willow oaks." | Wikipedia |
| Ground (USDA) | American beautyberry | "Look on ground that drains well, not soggy." | USDA PLANTS |
After Gemma runs, deterministic checks confirm the quote is really in the article and the plant itself is never named. They also check for banned words like "safe," "harmless," or "touch," make sure no hint exceeds 20 words total, and throw out anything that sends a kid toward any road, body of water, or a giant cliff.
USDA PLANTS fills in the gaps using its shade and soil ratings, size, season, and common signs. After both lists are joined, the code drops anything without a source as a safeguard, and then all hints for a specific plant are ranked by aspect, source, and rarity.
On the first full run, Claude graded a random sample of 300 picked hints against their quotes to determine overreach and spot inaccuracies: 94% were supported and none were unsafe after a recheck.
Claude grading Gemma could only get me so far, so I read a sample of the generated hints and graded them myself. 13 were off: 10 unsupported, 3 overreaching, so I tightened the rules, reran Gemma, and graded a fresh set. Only 3 of 48 in that run were off — a total of 6%. The stricter rules did cost plant coverage in the game, but 1,849 of 2,093 playable plants — or 88% — still have at least one hint (hint results).
This doesn't make the game error-free, but it helps reduce the number of errors a user may find from any location. I tested it in West Georgia, and the game works better inside the United States; however, it supports regions worldwide where iNaturalist has enough sightings.
Why Does Open Innovation Matter?
I wanted the game to work offline, for campers and out in the backwoods. That means I can also use the app when I go back home to the mountains, where cell phone signal is unreliable at best.
Offline functionality depends on a cached lookup in the user's hunt region. Parents have the ability to pre-load a cached list for all 12 months of a selected region from the grown-ups page to prep for offline use ahead of time.
!Wild Find's grown-ups page with the Cache my area button, which saves all 12 months of plants for the hunting area.
The only calls that leave the device when it's online go to iNaturalist.org without any user accounts or logins.
!Wild Find's offline banner: No signal. Using your saved hunt.
!Wild Find's offline message: This place needs signal once. After that, hunts here work without it.
I caught one potentially fatal problem to my privacy statement during testing. ONNX Runtime quietly opens a telemetry connection to Microsoft unless you explicitly turn it off. After that, model startup ran with zero open connections and a fresh hunt only ever talks to iNaturalist (privacy log).
Having access to open-weight models meant I could test different approaches without paying for every photo, bundle the models directly in the APK, and keep the entire recognition process on the phone. I didn't have to choose between asking kids to upload their pictures or paying for a hosted API every time somebody wanted to find a tree.
Prize Categories
Best Use of Gemma
My original plan was to use Gemma E2B, since the E4B I know runs hot on my device from my last Gemma project, Vestige. The problem with that is that I ultimately needed Gemma to function as a knowledge base, which it can't do without grounding (asked to describe the 20 most-seen plants near me, it got 4 right and 7 wrong enough to mislead).
My next idea was to just let it identify the plant, which was before I found the TinyCLIP and BioCLIP models. It performed terribly in early rounds of testing (2.3 to 2.9 seconds per photo once warm, when a live camera needs an answer about every 200 ms). So I found open-weight models that could do this job, which turned out to be the only thing that saved this game.
The next job I wanted Gemma to do was to help provide hints during gameplay. My idea was if we gave Gemma E2B a picture and what it was looking for, then it could tell the user "look to your right." That round of testing produced the same hint five times over (across two outdoor runs, every one said some version of "look for a big tree near the woods edge," whatever was in the picture).
Then I was left with rewriting text alone from a precompiled list at the cost of 0.7 seconds per run. What I got in return was the exact same phrase I gave it, minus a few words here and there, stacked on top of a long list of hallucinations from previous test runs. So, I decided to cut Gemma from the phone entirely.
Even though there's no active Gemma running in the app itself, I did use a local Gemma instance through Ollama to help build both the hints displayed for each plant and the hazard warnings that appear in the game. I've wasted hundreds of dollars using GPT, Claude, and Gemini APIs to do this sort of proof-of-concept work before this challenge, and I really wasn't interested in adding to that number.
Gemma also decided which plants made the contact-hazard list in the first place. It had to quote the exact Wikipedia sentence it relied on, and anything it wasn't sure about came to me for review before it could ship.
I tested both Gemma 12B and 26B models. 12B made up three different source quotes. 26B made up none and it ran faster on my laptop, anyway. The decision made itself, really.
The results ship with the APK as data and are used for lookups later.
It might not be the best use case, but it's a real one. It also shows you exactly where the lines are drawn between AI capability and wishful thinking. AI is not meant to perform a job I can accomplish in the codebase, nor should it be kept for a challenge when the cost exceeds the benefits. I decided to enter this category anyway, so I could show you exactly why this was the right call.
Future Enhancements
- Increase the real-find pass rate to > 90% without letting a toxic plant win.
- Improve hint coverage to include all plants by sourcing information outside of Wikipedia and USDA PLANTS.
- Shrink the APK and make installing it easier for people who aren't developers.
Ultimately, I'd rather a kid hear "keep looking" than win with poison ivy, so that's the line I drew throughout to keep things honest.
{% embed dev.to %}
---
🛡️ Picked by Me, Not the Plants
Claude, Codex, and Copilot all had a hand in this build, right up until Copilot ran out of credits and sat out the rest of the hunt. Leonardo painted the pictures, and Claude checked every number against the repo, dragged me back to the logs whenever my memory wandered off, and wrote this footer. The decisions, the 0.034 margin, and the barefoot test pilot are all mine.
21 reactions · 2 comments
Mentions
GoogleのEmbeddingGemma 2をOpenCLIP風に使えるPythonライブラリを作ってPyPIに公開してみた はじめに GoogleのEmbeddingGemma 2を、OpenCLIPに近いインターフェースで扱えるPythonライブラリ「openclip-gemma」を開発し、PyPIに公開しました。 GitHub: kotadymo/openclip-gemma PyPI: pypi.org pip install openclip-gemma openclip-gemmaは、EmbeddingGemma 2を使った画像・テキストの埋め込み生成を、create_model_and_transforms()、get_tokenizer()、encode_image()、encode_text()といった、OpenCLIPでおなじみのAPIで実行できるようにするライブラリです。 画像とテキストの類似度計算、画像検索、Zero-shot分類などの処理を、比較的シンプルなPythonコードで試せることを目指しています。 今回は、開発した背景、ライブラリの特徴、インストール方法、使い方、現時点での制約について紹介します。 追記 本ライブラリの名称「openclip-gemma」は、OpenCLIP風のAPIでEmbeddingGemma 2を利用できるという特徴から付けたものです。 本ライブラリは個人開発の非公式プロジェクトであり、Google、Google DeepMind、OpenCLIPプロジェクトとは提携・協力関係になく、公式に承認されたものでもありません。 既存プロジェクトの名称や商標への配慮から、今後ライブラリ名を変更する可能性があります。変更する場合は、GitHubで移行方法をご案内します。 1. なぜ作ったのか 本音 この記事はほぼすべてAIに書かせましたが、こちらに作った理由の本音を記載しておきます。 個人製作の中でよくCLIPを使用しています。特定のシーンを検出するのにチューニングに苦労しているのですが、先々週このEmbeddingGemma 2というモデルが登場しました。いろいろと調べていると、CLIPと同様の画像とテキストの比較ができるということがわかりました。しかし、CLIPとは異なる考えのもと作成されているので、CLIPと同様に使えるわけではありません。 なので、「CLIPと同じような感覚でEmbeddingGemma 2を扱えるライブラリがあったら便利じゃね？」という安易な考えから、作ってみることにしました。ちなみに、今回の開発ではコードの実装をほぼすべてAIに任せています。私自身、Pythonライブラリを公開するのは初めてです。 OpenCLIPの使いやすいAPI OpenCLIPは、画像とテキストを同じ埋め込み空間で扱うCLIP系モデルを利用するためのライブラリです。 例えば、モデルの初期化、画像の前処理、テキストのトークン化、埋め込み生成を次のような流れで実行できます。 import open_clip model, _, preprocess = open_clip.create_model_and_transforms( "ViT-B-32", pretrained="..." ) tokenizer = open_clip.get_tokenizer("ViT-B-32") このインターフェースは直感的で、画像とテキストを組み合わせた処理を実装しやすいと感じています。 一方で、EmbeddingGemma 2はOpenCLIPのモデルとしてそのまま扱えるわけではありません。 そこで、EmbeddingGemma 2の公式な推論処理を維持しながら、OpenCLIPに近い書き方で利用できないかと考えました。 EmbeddingGemma 2を利用したい EmbeddingGemma 2は、Googleの画像・テキスト対応の埋め込みモデルです。 画像とテキストの両方を768次元の特徴ベクトルに変換でき、ベクトル間の類似度を利用することで、画像と文章の関連性を計算できます。 例えば、次のような用途を想定できます。 - テキストから関連する画像を検索する - 画像に近い説明文を選択する - 複数の文章候補と画像の類似度を比較する - 画像とテキストを利用した検索システムを試作する そこで、OpenCLIPに似たAPIでEmbeddingGemma 2を扱える、推論専用のラッパーライブラリを作成しました。 2. openclip-gemmaの特徴 主な特徴は以下のとおりです。 | 項目 | 内容 | |---|---| | 対応モデル | Google EmbeddingGemma 2 | | 画像入力 | 対応 | | テキスト入力 | 対応 | | 日本語テキスト | 対応 | | 埋め込み次元 | 768次元 | | API | OpenCLIP風 | | バッチ推論 | 対応 | | Zero-shot類似度計算 | 対応 | | 推論エンジン | PyTorch | | 学習・Fine-tuning | 非対応 | | 動画・音声 | 本ライブラリでは非対応（モデル自体は対応） | 公式モデルの画像前処理、トークナイザー、プロンプト、プーリング、正規化の処理を維持するように実装しています。 また、モデル重みはパッケージに同梱していません。実行時にHugging Faceから取得します。 モデルには特定のリビジョンを指定しており、モデル側の更新によって意図せず推論結果が変化するリスクを抑えています。 なお、OpenCLIPの完全な互換実装ではありません。 OpenCLIPのすべてのモデルやAPIを実行できるわけではなく、EmbeddingGemma 2専用の推論ラッパーです。 3. インストール 現時点で検証済みの環境は以下です。 - Windows 11 x64 - Python 3.12 - CPU（FP32およびBF16） Linux、macOS、CUDAについては、v0.1.0時点では動作検証が完了していません。 まず仮想環境を作成します。 python -m venv .venv Windowsの場合は以下で有効化できます。 .\.venv\Scripts\Activate.ps1 続いてインストールします。 python -m pip install openclip-gemma==0.1.0 PyTorchなど、推論に必要な依存ライブラリもインストールされます。 初回のモデル読み込みでは、Hugging Faceからモデルファイルを取得するため、ネットワーク接続とダウンロード時間、ディスク容量が必要です。 4. 基本的な使い方 4.1 モデルの読み込み まず、EmbeddingGemma 2を読み込みます。 import torch import openclip_gemma device = "cpu" model, _, preprocess = ( openclip_gemma.create_model_and_transforms( "embeddinggemma-2", device=device, ) ) model.eval() create_model_and_transforms()は、OpenCLIPに近い形式でモデルと画像前処理関数を返します。 今回はCPUを明示しています。 4.2 画像を768次元のベクトルに変換する Pillowで画像を読み込み、画像の埋め込みを生成します。 from PIL import Image image = Image.open("sample.jpg").convert("RGB") image_tensor = ( preprocess(image) .unsqueeze(0) .to(device) ) with torch.inference_mode(): image_features = model.encode_image( image_tensor, normalize=True, ) print(image_features.shape) 出力例： torch.Size([1, 768]) 1枚の画像から768次元の特徴ベクトルが得られます。 ここで注意したいのは、画像に対してライブラリが返すpreprocessを使うことです。 EmbeddingGemma 2固有の画像前処理を想定しているため、別のCLIPモデル用に正規化したテンソルをそのまま渡すことはできません。 4.3 日本語テキストをベクトルに変換する 続いて、テキストの埋め込みを生成します。 tokenizer = openclip_gemma.get_tokenizer( "embeddinggemma-2" ) texts = [ "机の上に置かれたノートパソコン", "公園を走る犬", "青空の下にある建物", ] tokens = tokenizer(texts).to(device) with torch.inference_mode(): text_features = model.encode_text( tokens, normalize=True, ) print(text_features.shape) 出力例： torch.Size([3, 768]) 3つの日本語テキストそれぞれについて、768次元の特徴ベクトルが得られます。 get_tokenizer()を利用することで、モデルが必要とするトークン化を実行できます。 5. 画像と文章の類似度を計算する ここが今回のライブラリで特に試していただきたい部分です。 画像と複数の説明文を比較し、どの文章が画像に近いかを計算します。 import torch from PIL import Image import openclip_gemma device = "cpu" # モデルの読み込み model, _, preprocess = ( openclip_gemma.create_model_and_transforms( "embeddinggemma-2", device=device, ) ) model.eval() # 画像の読み込み image = Image.open("sample.jpg").convert("RGB") image_tensor = preprocess(image).unsqueeze(0).to(device) # 比較したいテキスト candidate_texts = [ "机の上に置かれたノートパソコン", "公園を走る犬", "青空の下にある建物", ] # 分類タスク用の公式プロンプトを使用 tokenizer = openclip_gemma.get_tokenizer( "embeddinggemma-2", prompt_name="Classification", ) text_tokens = tokenizer(candidate_texts).to(device) # 埋め込み生成と類似度計算 with torch.inference_mode(): image_features = model.encode_image( image_tensor, normalize=True, ) text_features = model.encode_text( text_tokens, normalize=True, ) similarities = image_features @ text_features.T # 結果表示 scores = similarities[0].tolist() for text, score in zip(candidate_texts, scores): print(f"{text}: {score:.4f}") best_index = similarities[0].argmax().item() print("最も類似度が高い文章:") print(candidate_texts[best_index]) image_features @ text_features.Tは、正規化された画像・テキストの特徴ベクトル間の内積を計算しています。 これはコサイン類似度として利用できます。 類似度が高いほど、その画像と文章がモデルの埋め込み空間上で近いことを意味します。 ただし、類似度は分類確率ではありません。 例えばスコアが0.75であっても、「75%の確率で正解」という意味ではありません。 また、最も高いスコアの文章が常に正しいとは限りません。用途ごとに候補文の設計や精度評価が必要です。 6. OpenCLIPからの移行イメージ 既存のOpenCLIPを利用した推論コードでは、次のような書き方をしていることがあります。 import open_clip model, _, preprocess = ( open_clip.create_model_and_transforms( "ViT-B-32", pretrained="...", ) ) tokenizer = open_clip.get_tokenizer("ViT-B-32") openclip-gemmaの場合は次のようになります。 import openclip_gemma as open_clip model, _, preprocess = ( open_clip.create_model_and_transforms( "embeddinggemma-2", device="cpu", ) ) tokenizer = open_clip.get_tokenizer( "embeddinggemma-2" ) 初期化部分の構造が似ていることが分かると思います。 さらに、次のAPIも利用できます。 model.encode_image(...) model.encode_text(...) このため、画像とテキストを別々にエンコードして類似度を計算するような処理を、比較的近い構成で記述できます。 ただし、モデルごとの前処理、トークン化、プロンプト、埋め込み特性は異なります。 既存のOpenCLIPコードをimport文だけ変更すれば必ず動くというものではありません。 詳細は互換性レポートに整理しています。 7. 実装・精度検証について 今回の開発では、APIを似せるだけではなく、公式の推論結果と大きく乖離しないことを重要視しました。 そのため、公式のSentence Transformers経由の推論結果と、openclip-gemmaの出力を比較しています。 Windows 11、Python 3.12、CPU FP32で測定した最大絶対誤差は、以下のとおりでした。 | 比較対象 | 最大絶対誤差 | |---|---:| | 画像埋め込み | 約2.74 × 10⁻⁷ | | テキスト埋め込み | 約2.81 × 10⁻⁶ | | 画像・テキスト類似度 | 約1.19 × 10⁻⁶ | 特定のモデルリビジョンとテスト入力・実行環境における比較結果ですが、公式の推論経路と非常に近い値が得られています。 また、リリース前には以下を確認しています。 - pytest：75件成功、9件スキップ - 実モデルを使ったCPU統合テスト：7件成功、CUDA専用の2件をスキップ - Ruffによる静的チェック - wheel・sdistのビルドと内容検査 - TestPyPIからクリーン環境へのインストール - 画像・日本語テキストを利用した実モデル推論 - GitHub ActionsによるCIとPyPIへの自動公開 ただし、これらは推論処理の整合性や配布の正常動作を確認するものであり、個別の画像分類・検索タスクにおける精度を保証するものではありません。 8. 現時点での制約 v0.1.0は初期リリースのため、機能を限定しています。 特に次の点には注意してください。 対応モデルはEmbeddingGemma 2のみです。 任意のCLIPモデルを読み込むことはできません。 推論専用です。 学習やFine-tuningには対応していません。 動画・音声には対応していません。 現在は静止画像とテキストの埋め込み生成を対象としています。 CUDA、Linux、macOSは未検証です。 CUDA向けのコードは実装していますが、GPU実機での動作確認は今後の課題です。 OpenCLIPの完全互換ではありません。 forward()、get_logits()など、一部のOpenCLIPで使われる機能は提供していません。また、OpenCLIPのlogit_scaleを再現するものではありません。 9. 今後について 今後は、実際の利用状況を見ながら、対応環境の拡大や使い勝手の改善を検討したいと考えています。 例えば、CUDA環境での実機検証、Linux・macOSでの動作確認、サンプルコードやドキュメントの充実などが考えられます。 初期バージョンのため、不具合や改善点があればGitHub Issuesで報告いただけるとうれしいです。 github.com/kotadymo/openclip-gemma/issues おわりに 今回は、EmbeddingGemma 2をOpenCLIP風のAPIで扱えるPythonライブラリ「openclip-gemma」を紹介しました。 画像とテキストの埋め込み生成や類似度計算を、できるだけシンプルなコードで実行できるようにしたライブラリです。 EmbeddingGemma 2を試してみたい方や、OpenCLIPに近い書き方で画像・テキスト埋め込みを扱いたい方の参考になれば幸いです。 まだv0.1.0の初期段階ですが、ぜひ試してみてください。 GitHub: kotadymo/openclip-gemma PyPI: pypi.org なお、本ライブラリは個人開発の非公式プロジェクトであり、Google、Google DeepMind、OpenCLIPの公式実装・公式配布物ではありません。ライブラリ本体はApache-2.0で公開しています。モデル重みは同梱しておらず、利用時にはGoogleのモデルに適用される別途の利用条件をご確認ください。
Side Quests Grow on Trees (Leave Them Where They Grow) 🌿 --- description: A kids' plant treasure hunt for Android with on-device BioCLIP and TinyCLIP, Gemma 4 hints built locally, and 0 of 267 toxic test photos passing as a match. --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass TL;DR - Wild Find is a native Android treasure hunt for kids 8 and up, built for (and tested by) my family, that sends kids after real plants common to their area. - Two open-weight models run on the phone, BioCLIP 2.5 Mobile and TinyCLIP, and no images ever leave the device. - Gemma 4 26B runs locally in the build pipeline to precompile hints and hazard warnings, and every win still comes down to deterministic code: 0 of 267 toxic test photos passed as a match. - 🎵 I love assigning a theme song to my builds, just because it's fun. This one is Pinecones by Bug Hunter, and it's fabulous! - 🎬 Watch the demo · 💻 Browse the repo · 📱 APK releases What I Built The idea for this game came from a family powwow, where every idea built off the last until I was satisfied that we had outlined something I could realistically build. Wild Find is a treasure hunt game designed for kids 8 years and up to play with their parents. Even though my son is much older than that, I still had a lot of fun building and testing! !Screenshot Wild Find home screen. This is not a quick game. It's designed to keep kids, both young and old, outside looking for plants as long as they can stand it. The app gives you three randomly selected plants for a region that is set by an adult user based on location or picked from a map. The kid is then tasked to find one of three plants common to their specific area of the world. When they think they've found one, they capture a picture without picking the plant, or they can skip an extra hard one. The game is finished when you find all three plants from your list. !Wild Find's hunt list for Georgia in October: blue mistflower, eastern redbud, and American beautyberry, each with a short description. My son and sister both played the game without any instruction from me. We all learned that the previously deferred in-game hints were a necessity to prevent users from getting frustrated by the game. Also, just because my region contains those plants does not mean that my yard does. So I added in skip functionality for each plant to keep it kid-friendly. If the app flags a potentially hazardous plant, it warns the user not to touch it — which is a general safety rule in the game, anyway. There are 85 total hazards: seven in North America that are always on, like poison ivy, oak, and sumac, and another 78 that are only on when iNaturalist says they're nearby. !Wild Find's hazard warning over a poison ivy photo: That might be a plant we leave extra space around. Its oil causes an itchy and painful rash. Step back and look somewhere else. I'm most proud of the latency numbers I was able to accomplish with two open-weight models: BioCLIP 2.5 Mobile, a plant-focused model distilled from the larger BioCLIP 2.5, and TinyCLIP, a tiny general-purpose model that checks whether there's a plant in the frame at all. Together, they take a median of 189 ms per frame on my S24 Ultra. As always, I built this app to be accessible from the start: big touch targets, labels for TalkBack, no color-only signals, animations that follow the reduced-motion settings, and hints and warnings that are still there when text is set to 200% size. The app rounds the user's rough coordinates to the nearest whole degree before sending them to iNaturalist, which returns a list of regional plants, each with a short description of its type and up to three hints prebuilt from Wikipedia and USDA PLANTS. No photos or saved gameplay progress leave the device. iNaturalist also sees the request's IP address, as disclosed on the grown-ups page. Demo {% embed youtu.be %} 🎵 Theme song: Pinecones by Bug Hunter > 📱 Want to play it yourself? Grab the Android APK (Android 11+) from the v1.0.0 release. Code {% embed anchildress1/wild-find %} > ⚖️ This project is licensed under MIT. I build with AI, including Claude, Codex, and Copilot — which ran out of credits before I had finished the build. There are plenty of distinct design decisions you can see in the code throughout. First, I don't want a toxic or hazardous plant to win a hunt. I guard against this with a "blocker," where every local toxic or hazard species is checked in the photo capture. The best match must beat them all by 0.034 when the full frame is recognized as a plant, or 0.048 when only the circle is recognized. A blocker also scores on the circle alone, so a toxic plant that only fills the circle can't hide behind the rest of the photo. I tested this until 0 of 267 toxic plant photos passed, while real finds went from 97 to 108 of 198. override fun score(reticle: FloatArray, full: FloatArray?): GoalScore { fun s(row: Int) = if (full == null) { table.dot(row, reticle) } else { (table.dot(row, reticle) + table.dot(row, full)) / 2 } val margin = if (full == null) MARGIN else FULL_MARGIN val scores = eligible.map(::s) val best = scores.max() val top = eligible[scores.indexOf(best)] // A toxic plant that fills only the circle must not be diluted by the full frame, so a blocker scores the // larger of its mean and its reticle cosine. fun guard(row: Int) = if (full == null) s(row) else maxOf(s(row), table.dot(row, reticle)) val blocker = blockers.maxOfOrNull(::guard) ?: Double.NEGATIVE_INFINITY val score = s(target) // A tie for top-1 is no pass, whichever rows tie. val met = scores.count { it == best } == 1 && genus[top] == genus[target] && best - blocker >= margin return GoalScore(met, score, 1 + scores.count { it > score }) } /** * The reticle-only margin, for a frame whose full frame isn't a plant: the smallest round margin that let 0 of * 180 local toxic photos pass on Oct 7 (a poison ivy photo read as beautyberry won by 0.0477). */ const val MARGIN = 0.048 /** * The margin on the reticle and full-frame mean (day-5 rule R1h): the 0.024 tune boundary plus 0.01 headroom. * With blockers floored at their reticle cosine it lets 0 of 267 toxic photos pass, with the closest toxic * photo 0.020 under the margin, and raises target passes from 97 to 108 of 198 (`blocker_guard.log`). */ const val FULL_MARGIN = 0.034 Second, according to BioCLIP's model card, it matches an image against a fixed table of known species, and it's strongest at the genus level. The same card says "Do not use this for toxicity or edibility decisions," which is why the toxicity flags come from Wikipedia and USDA text, not the model. So the app takes the full local list, asks BioCLIP to rank the capture image against each local plant species, and then deterministic checks run to determine if a match was made or not. The code uses BioCLIP's scores for each plant, averaged across the circle and the whole photo when TinyCLIP sees a plant in the whole photo (for example, how much does this look like water oak?), to determine the top species and whether that top species is in the target's genus (because this is only the easy level; future iterations would increase the difficulty). The 0.034 margin originally came from an earlier rule's 0.024 tuning boundary plus 0.01 of headroom. After adding the circle-only blocker guard, I tested the rule again. Lowering the margin to 0.011 allowed two toxic photos through in the fresh test set, so I kept 0.034. That costs some legitimate finds, but the closest toxic test photo now falls 0.020 below the margin instead of just 0.0028. A single matching frame still isn't a win, either. The app needs three matching frames in a row before it counts a find. The Day 5 results have every rule I tried and every photo. fun of(sightings: List<Sighting>): LocalList { val floor = max(MIN_SIGHTINGS.toDouble(), SHARE * sightings.sumOf { it.count }) val byRow = sightings.mapNotNull { s -> rowOf(s.scientific)?.let { it to s } }.groupBy({ it.first }, { it.second }) val counted = byRow.mapValues { (_, names) -> names.sumOf { it.count } }.filterValues { it >= floor } val eligible = counted.mapNotNull { (row, count) -> val common = byRow.getValue(row).sortedByDescending { it.count } .firstNotNullOfOrNull { it.common?.trim()?.takeIf(::isKidName) } if (table[row].playable && common != null) Eligible(row, common, count) else null }.sortedWith(compareByDescending<Eligible> { it.count }.thenBy { it.row }) // Any sighting blocks: the floor-only blockers let 87 of 180 toxic photos pass as some target on Oct 7. val blockers = byRow.keys.filterNot { table[it].playable }.sorted().toIntArray() // A hunt takes one target per genus, so three species of two genera still can't fill it; a row held back from // picks can't fill a slot either. val genera = eligible.filter { table[it.row].target }.distinctBy { table[it.row].genus }.size return LocalList(eligible, blockers, needsWiden = genera < MIN_TARGETS) } Also, there were four plants BioCLIP couldn't reliably recognize in testing: sweetgum, tulip tree, persimmon, and winged sumac. So I pulled them all out of the hunt. They will still count when the app compares plants, though, so nothing sneaks through as one of those. There are plenty of other examples in the repo, too. Check out these pages, if you're curious. | File | What it does | Why it's code, not AI | |---|---|---| | Verify.kt | The verdict table: every message a kid sees after a capture, first matching row wins | One readable rule table decides the star, with no model judgment | | HazardCheck.kt | Warns on 7 fixed hazards, including poison ivy, oak, and sumac, plus 78 regional ones, using local iNat sightings | The warning is a fixed cutoff that was tested on 52 hazard photos, not a model's opinion | | toxicity.py | Flags toxic plants from Wikipedia sentences and USDA ratings | CLIP models scored toxicity at coin-flip odds (AUC 0.42–0.65), so text matching does the job | | assets.py | Refuses to build if any plant is missing its toxicity flag, and never gives a toxic or hazard plant a hint | The build fails closed: a safety gap stops the APK, it doesn't ship it | | RegionKey.kt | Rounds the location to a whole degree before anything leaves the phone | Privacy for kids comes from a single rounding function | I tested the hazard warning separately, too. It caught 48 of 52 hazard photos and only warned on 3 of 253 regular plants by mistake. Worldwide, using a fresh set of images, it caught 41 of 52, which isn't perfect. And it's exactly why the game never tells a kid that a plant is safe to touch (hazard results). Also, this build includes 314 Kotlin unit tests, 350 Python tests, and another 120 tests that run on device (67 device and 53 end-to-end). The automated checks pass, and it survived several rounds of manual testing on different devices, as proven by my son playing barefoot at the edge of our property. !My son playing Wild Find barefoot at the edge of our property How I Built It The app is separated into two distinct sections: the build pipeline and the app itself that runs on device. | Open-source AI | Type | Runs on | Job | |---|---|---|---| | BioCLIP 2.5 Mobile | Open-weight model | Phone | Scores each capture against the local plant list | | TinyCLIP ViT-8M | Open-weight model (MIT) | Phone | Checks there's a plant in the frame at all | | ONNX Runtime | Open-source inference runtime | Phone | Runs both models offline, with its telemetry switched off | | BioCLIP 2.5 (ViT-H) + OpenCLIP | Open-weight model + framework | Laptop, build time | Writes the tutorial vectors and the hazard rows BioCLIP's table is missing | | Gemma 4 26B | Open-weight model | Laptop, build time | Writes the hints and hazard warnings | | Ollama | Local inference | Laptop, build time | Serves Gemma with no cloud API calls | I looked into training a new model based on BioCLIP that could identify non-plants or poisonous plants, and could return hints inside the game. Based on research and model documentation already published on Hugging Face (Gemma 4 E2B, BioCLIP 2.5 Mobile, BioCLIP 2), I decided that would probably waste a lot of time and still not get me the results that I wanted. Early testing had already proved a larger model would slow down gameplay. Gemma 4 E2B is a 2.6 GB file next to BioCLIP Mobile's 47 MB, and it took 2.3 to 2.9 seconds per photo, while BioCLIP and TinyCLIP together take a median of 189 ms per frame. Instead, I chose to precompile a list of 4,272 plants, including the 2,093 not flagged as toxic or hazardous, and build that list into the released APK. First the build pipeline runs Gemma 26B against plant Wikipedia articles to determine hints based on hint type. | Hint type | Plant | Hint | Source | |---|---|---|---| | Place | Blue mistflower | "Look in sandy woodlands and clearings." | Wikipedia | | Light | Water oak | "It does not like shade." | Wikipedia | | Ground | Christmas fern | "It grows in moist habitats." | Wikipedia | | Nearby | American sweetgum | "It grows near willow oaks." | Wikipedia | | Ground (USDA) | American beautyberry | "Look on ground that drains well, not soggy." | USDA PLANTS | After Gemma runs, deterministic checks confirm the quote is really in the article and the plant itself is never named. They also check for banned words like "safe," "harmless," or "touch," make sure no hint exceeds 20 words total, and throw out anything that sends a kid toward any road, body of water, or a giant cliff. USDA PLANTS fills in the gaps using its shade and soil ratings, size, season, and common signs. After both lists are joined, the code drops anything without a source as a safeguard, and then all hints for a specific plant are ranked by aspect, source, and rarity. On the first full run, Claude graded a random sample of 300 picked hints against their quotes to determine overreach and spot inaccuracies: 94% were supported and none were unsafe after a recheck. Claude grading Gemma could only get me so far, so I read a sample of the generated hints and graded them myself. 13 were off: 10 unsupported, 3 overreaching, so I tightened the rules, reran Gemma, and graded a fresh set. Only 3 of 48 in that run were off — a total of 6%. The stricter rules did cost plant coverage in the game, but 1,849 of 2,093 playable plants — or 88% — still have at least one hint (hint results). This doesn't make the game error-free, but it helps reduce the number of errors a user may find from any location. I tested it in West Georgia, and the game works better inside the United States; however, it supports regions worldwide where iNaturalist has enough sightings. Why Does Open Innovation Matter? I wanted the game to work offline, for campers and out in the backwoods. That means I can also use the app when I go back home to the mountains, where cell phone signal is unreliable at best. Offline functionality depends on a cached lookup in the user's hunt region. Parents have the ability to pre-load a cached list for all 12 months of a selected region from the grown-ups page to prep for offline use ahead of time. !Wild Find's grown-ups page with the Cache my area button, which saves all 12 months of plants for the hunting area. The only calls that leave the device when it's online go to iNaturalist.org without any user accounts or logins. !Wild Find's offline banner: No signal. Using your saved hunt. !Wild Find's offline message: This place needs signal once. After that, hunts here work without it. I caught one potentially fatal problem to my privacy statement during testing. ONNX Runtime quietly opens a telemetry connection to Microsoft unless you explicitly turn it off. After that, model startup ran with zero open connections and a fresh hunt only ever talks to iNaturalist (privacy log). Having access to open-weight models meant I could test different approaches without paying for every photo, bundle the models directly in the APK, and keep the entire recognition process on the phone. I didn't have to choose between asking kids to upload their pictures or paying for a hosted API every time somebody wanted to find a tree. Prize Categories Best Use of Gemma My original plan was to use Gemma E2B, since the E4B I know runs hot on my device from my last Gemma project, Vestige. The problem with that is that I ultimately needed Gemma to function as a knowledge base, which it can't do without grounding (asked to describe the 20 most-seen plants near me, it got 4 right and 7 wrong enough to mislead). My next idea was to just let it identify the plant, which was before I found the TinyCLIP and BioCLIP models. It performed terribly in early rounds of testing (2.3 to 2.9 seconds per photo once warm, when a live camera needs an answer about every 200 ms). So I found open-weight models that could do this job, which turned out to be the only thing that saved this game. The next job I wanted Gemma to do was to help provide hints during gameplay. My idea was if we gave Gemma E2B a picture and what it was looking for, then it could tell the user "look to your right." That round of testing produced the same hint five times over (across two outdoor runs, every one said some version of "look for a big tree near the woods edge," whatever was in the picture). Then I was left with rewriting text alone from a precompiled list at the cost of 0.7 seconds per run. What I got in return was the exact same phrase I gave it, minus a few words here and there, stacked on top of a long list of hallucinations from previous test runs. So, I decided to cut Gemma from the phone entirely. Even though there's no active Gemma running in the app itself, I did use a local Gemma instance through Ollama to help build both the hints displayed for each plant and the hazard warnings that appear in the game. I've wasted hundreds of dollars using GPT, Claude, and Gemini APIs to do this sort of proof-of-concept work before this challenge, and I really wasn't interested in adding to that number. Gemma also decided which plants made the contact-hazard list in the first place. It had to quote the exact Wikipedia sentence it relied on, and anything it wasn't sure about came to me for review before it could ship. I tested both Gemma 12B and 26B models. 12B made up three different source quotes. 26B made up none and it ran faster on my laptop, anyway. The decision made itself, really. The results ship with the APK as data and are used for lookups later. It might not be the best use case, but it's a real one. It also shows you exactly where the lines are drawn between AI capability and wishful thinking. AI is not meant to perform a job I can accomplish in the codebase, nor should it be kept for a challenge when the cost exceeds the benefits. I decided to enter this category anyway, so I could show you exactly why this was the right call. Future Enhancements - Increase the real-find pass rate to > 90% without letting a toxic plant win. - Improve hint coverage to include all plants by sourcing information outside of Wikipedia and USDA PLANTS. - Shrink the APK and make installing it easier for people who aren't developers. Ultimately, I'd rather a kid hear "keep looking" than win with poison ivy, so that's the line I drew throughout to keep things honest. {% embed dev.to %} --- 🛡️ Picked by Me, Not the Plants Claude, Codex, and Copilot all had a hand in this build, right up until Copilot ran out of credits and sat out the rest of the hunt. Leonardo painted the pictures, and Claude checked every number against the repo, dragged me back to the logs whenever my memory wandered off, and wrote this footer. The decisions, the 0.034 margin, and the barefoot test pilot are all mine.
あとは openclip_package = os.environ.get('OPENCLIP_PACKAGE', "github.com/mlfoundations/open_clip/archive/bb6e83…") を openclip_package = os.environ.get('OPENCLIP_PACKAGE', "open-clip-torch") に変更して、もう一度webui-user.bat を動かせばいけた