How to Build Your Own Google Antigravity Plugin: Packaging Skills, MCP Servers, Subagents, Rules, and Hooks
---
title: "How to Build Your Own Google Antigravity Plugin: Packaging Skills, MCP Servers, Subagents, Rules, and Hooks"
published: true
description: "A practical, step-by-step engineering guide to packaging team workflows, MCP tools, custom agents, and deterministic guardrails into a single versioned Antigravity plugin."
tags: antigravity, ai, googlecloud, devtools
---
> "Declarative rules define how an engineering team works; plugins make those standards portable across every repository and developer machine."
When teams first adopt coding agents, customization usually starts as a single AGENTS.md file at the root of a repository. A week later, someone adds a slash-command runbook under .agents/skills/. Two weeks later, a platform engineer wires up a Model Context Protocol (MCP) server and writes a shell script to stop the agent from running terraform apply against production.
In a single repository, loose files in .agents/ work fine. Across twenty microservices or a dozen infrastructure repositories, copy-pasting those files turns into configuration drift. One repo has an outdated IAM audit skill, another is missing the PreToolUse safety hook, and new hires spend their first afternoon debugging hardcoded paths in mcp_config.json.
Google Antigravity Plugins solve this packaging problem. A plugin bundles Skills, MCP servers, Custom Agents (Subagents), Rules, and Lifecycle Hooks into a single versioned directory anchored by a plugin.json manifest. You can validate that directory locally with agy plugin validate --strict, share it via GitHub, drop it into a workspace, or publish it to the Antigravity Marketplace.
In this guide, we will dissect how the Antigravity plugin runtime works under the hood and build a complete, production-ready plugin from scratch: gcp-terraform-kit.
---
1. Loose Workspace Files vs. Packaged Plugins
Before writing our manifest, let us look at what changes when you move from ad-hoc .agents/ folders to a versioned plugin.
| Dimension | Loose .agents/ Files | Antigravity Plugin (plugin.json) |
| :--- | :--- | :--- |
| Distribution | Manual copy-paste across repositories | Single install via agy plugin install (local, GitHub URL, or Marketplace) |
| Versioning | Tracked per-repo in Git commits | Semantic versioning (MAJOR.MINOR.PATCH) in plugin.json |
| Tool Namespacing | Global names; prone to collisions | Automatic namespacing (<plugin-name>_<server-name>) |
| Filesystem Portability | Often relies on machine-specific paths | Built-in ${PLUGIN_ROOT} and ${PLUGIN_DATA} runtime variables |
| Credential Onboarding | Manual export in shell profiles | Declarative variables prompt dialog or browser OAuth 2.0 flow |
| Validation | Fails at runtime if YAML/JSON is malformed | Static verification via agy plugin validate --strict |
---
2. Anatomy of an Antigravity Plugin
An Antigravity plugin is a standard directory containing a plugin.json manifest at its root (or inside .antigravity-plugin/plugin.json if you maintain a multi-editor repository alongside .claude-plugin/ or .cursor-plugin/).
!Anatomy of a Google Antigravity Plugin
A plugin can bundle any combination of five customization primitives:
1. Skills (skills/<skill-name>/SKILL.md): On-demand procedural runbooks, helper scripts, and reference manuals invoked automatically when a prompt matches the skill description or explicitly via /<skill-name>.
2. MCP Servers (mcp_config.json): Local stdio processes or remote Streamable HTTP endpoints that expose external tools to the agent.
3. Custom Agents & Subagents (agents/<agent-name>.md): Specialist personas with isolated context windows, dedicated model tiers (pro, flash, inherit), and restricted tool allowlists.
4. Rules (rules/AGENTS.md and rules/<rule-name>.md): Always-on architectural guardrails or file-glob-triggered coding conventions.
5. Lifecycle Hooks (hooks.json): Synchronous scripts that intercept SessionStart, PreInvocation, PreToolUse, PostToolUse, and Stop events over stdin/stdout.
How Progressive Disclosure Protects Your Context Window
A common mistake when building agent extensions is stuffing 40 pages of documentation into an always-on system prompt. That burns tokens on every turn and degrades model attention.
!Progressive Disclosure & Lifecycle Execution Loop
Antigravity avoids context bloat through progressive disclosure:
- At session startup, the runtime loads rules/AGENTS.md, always_on rules, and only the name and description frontmatter of your bundled skills and subagents.
- During the session, when a developer opens a .tf file, Antigravity injects rules configured with trigger: glob matching **/*.tf.
- When a task matches a skill, the agent reads the full SKILL.md body and any linked files in references/ on demand.
- When heavy analysis is required, the primary agent delegates to a bundled subagent in a separate conversation trajectory, keeping raw file dumps out of your main chat history.
---
3. Hands-On Tutorial: Building gcp-terraform-kit
Let us build a real plugin for a Google Cloud platform engineering team. Our gcp-terraform-kit plugin will:
1. Enforce Google Cloud Terraform security conventions whenever .tf files are edited (Rules).
2. Provide an /audit-module workflow with a reference checklist (Skills).
3. Connect the official Terraform MCP server and a local state-inspector script using portable path variables (MCP Servers).
4. Ship a read-only iam-reviewer specialist agent (Subagents).
5. Block terraform apply and terraform destroy at the tool execution boundary (Lifecycle Hooks).
Start by creating the directory tree:
mkdir -p gcp-terraform-kit/{assets,scripts,skills/audit-module/references,agents,rules}
---
Step 1: Create the Manifest (plugin.json)
Every plugin requires plugin.json. Including the $schema key gives you autocomplete and inline validation in VS Code, JetBrains IDEs, and the Antigravity IDE.
// gcp-terraform-kit/plugin.json
{
"$schema": "https://antigravity.google/schemas/v1/plugin.json",
"name": "gcp-terraform-kit",
"displayName": "GCP Terraform Kit",
"version": "1.0.0",
"description": "Google Cloud Terraform guardrails, module audit workflows, IAM reviewer subagent, and apply protection hooks.",
"logo": "assets/logo.svg",
"suggestedPrompts": [
"Audit the current Terraform module for CIS Google Cloud Benchmark gaps",
"Review IAM bindings in this directory for least-privilege violations",
"Validate and test all .tf files in the current workspace"
],
"author": {
"name": "Carlos Barbero",
"url": "https://github.com/carlosrgomes"
},
"homepage": "https://github.com/carlosrgomes/gcp-terraform-kit",
"repository": "https://github.com/carlosrgomes/gcp-terraform-kit",
"license": "Apache-2.0",
"keywords": [
"terraform",
"gcp",
"google-cloud",
"iac",
"security"
]
}
> Important — Naming Rules for Strict Validation:
> * name must be lowercase kebab-case (^[a-z0-9]+(-[a-z0-9]+)*$). It is used to namespace your bundled MCP servers (gcp-terraform-kit_<server>).
> * displayName must not start with "Antigravity", "Gemini", "Claude", or "Cursor", and should omit suffixes like "Plugin" or "MCP".
> * suggestedPrompts accepts up to 3 starter prompts that render as clickable chips on the plugin detail page in the Antigravity App.
Let us also add a simple square SVG icon at assets/logo.svg (Marketplace plugins require a 1:1 square icon with ~10% padding):
<!-- gcp-terraform-kit/assets/logo.svg -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128" width="128" height="128">
<rect x="12" y="12" width="104" height="104" rx="22" fill="#0E1E40" stroke="#4CC9F0" stroke-width="4"/>
<path d="M40 36L62 48V72L40 60V36ZM66 50L88 38V62L66 74V50ZM40 66L62 78V102L40 90V66Z" fill="#4CC9F0"/>
</svg>
---
Step 2: Author Rules (rules/AGENTS.md and Glob Rules)
Plugins support two flavors of rules inside rules/:
1. rules/AGENTS.md (or rules/GEMINI.md): Loaded unconditionally whenever the plugin is enabled. No YAML frontmatter is needed.
2. Named rule files (rules/<rule-name>.md): Configured via YAML frontmatter with a trigger mode (always_on, model_decision, glob, or manual).
Let us create a short rules/AGENTS.md for global plugin behavior, and put our HCL-specific rules in rules/hcl-style.md using trigger: glob so they only consume context tokens when .tf or .tfvars files are active:
<!-- gcp-terraform-kit/rules/AGENTS.md -->
# GCP Terraform Kit — Core Policy
When working in repositories that contain Terraform configurations for Google Cloud:
- Delegate IAM and security reviews to the `iam-reviewer` subagent before proposing changes to `google_*_iam_*` resources.
- Never execute state-mutating commands (`terraform apply`, `terraform destroy`, `terraform state rm`) directly.
Now create the glob-scoped rule in rules/hcl-style.md:
---
trigger: glob
globs: "**/*.tf, **/*.tfvars, **/*.tftest.hcl"
description: Google Cloud Terraform coding and security standards
---
# Google Cloud Terraform Standards
1. **Provider Version Pinning**: Every root module and reusable module must constrain `google >= 6.0, < 7.0` in `versions.tf`.
2. **Zero Static Keys**: Never create `google_service_account_key` resources. Use Workload Identity Federation or Service Account Impersonation.
3. **Authoritative vs. Non-Authoritative IAM**: Prefer `google_project_iam_member` over `google_project_iam_policy` or `google_project_iam_binding` to avoid wiping existing role bindings.
4. **Labels & Encryption**: Every `google_storage_bucket`, `google_sql_database_instance`, and `google_bigquery_dataset` must expose `labels = var.labels` and an optional `kms_key_name` variable for CMEK.
> Warning: In rules/<rule-name>.md, the globs frontmatter field must be a comma-separated string ("**/*.tf, **/*.tfvars"), not a YAML list (["**/*.tf"]). Passing a YAML array will fail agy plugin validate --strict.
---
Step 3: Package an On-Demand Skill (skills/audit-module/SKILL.md)
Skills teach the agent repeatable multi-step procedures. Two rules govern plugin skills:
- The name field in SKILL.md frontmatter must exactly match the parent folder name (skills/audit-module/SKILL.md -> name: audit-module).
- Inside SKILL.md, reference helper scripts and docs using relative paths from the skill folder (references/cis-checklist.md).
---
name: audit-module
description: >-
Run a structural, syntactic, and CIS Google Cloud security audit on a
Terraform module. Use when the user asks to audit, review, or validate
a Terraform module, or invokes /audit-module.
---
# Terraform Module Audit Workflow
Follow these steps in order when auditing a Terraform directory:
1. **Static Validation**:
Run `terraform fmt -check -recursive` and `terraform validate` in the target directory.
2. **Checklist Verification**:
Read [references/cis-checklist.md](references/cis-checklist.md) and evaluate every `.tf` file against the controls.
3. **Subagent IAM Review**:
If the module declares any `google_*_iam_*` or `google_service_account` resources, delegate a deep check to the `iam-reviewer` subagent.
4. **Structured Report**:
Output a markdown table with columns: `Resource`, `Severity`, `Rule Violated`, and `Recommended Fix`.
Let us create the companion reference file so the skill stays concise:
<!-- gcp-terraform-kit/skills/audit-module/references/cis-checklist.md -->
# CIS Google Cloud Benchmark — Terraform Quick Checks
- **CIS 1.4**: Ensure no `google_service_account_key` resources exist.
- **CIS 5.1**: Ensure `google_storage_bucket` sets `uniform_bucket_level_access = true` and `public_access_prevention = "enforced"`.
- **CIS 6.2**: Ensure `google_sql_database_instance` enables `ssl_mode = "ENCRYPTED_ONLY"` and `deletion_protection = true`.
- **CIS 3.6**: Ensure `google_compute_firewall` never allows `0.0.0.0/0` on ports `22` or `3389`.
---
Step 4: Configure a Working MCP Server (mcp_config.json & scripts/state-mcp.mjs)
When a plugin is installed, Antigravity unpacks it into a read-only directory (~/.gemini/config/plugins/<plugin-name>) that gets replaced on upgrades. Because you never know the user's home directory path in advance, Antigravity provides three runtime variables for mcp_config.json:
- ${PLUGIN_ROOT}: Resolves to the read-only installation directory of the plugin.
- ${PLUGIN_DATA}: Resolves to a persistent, writable directory created specifically for your plugin outside the workspace (ideal for caches or audit logs).
- ${VARIABLE_NAME}: Interpolates user-prompted secrets declared in mcpServers.<server>.variables.
Let us configure a self-contained local Node.js MCP server (state-inspector) in mcp_config.json that uses ${PLUGIN_ROOT} and ${PLUGIN_DATA} so the plugin activates immediately upon installation:
// gcp-terraform-kit/mcp_config.json
{
"mcpServers": {
"state-inspector": {
"command": "node",
"args": [
"${PLUGIN_ROOT}/scripts/state-mcp.mjs",
"--cache-dir",
"${PLUGIN_DATA}/cache"
]
}
}
}
Now create the zero-dependency MCP stdio server in scripts/state-mcp.mjs. It implements the JSON-RPC 2.0 initialize, tools/list, and tools/call methods, exposing an inspect_terraform_workspace tool (namespaced at runtime as gcp-terraform-kit_state-inspector) that scans .tf files and caches the inspection report inside ${PLUGIN_DATA}:
#!/usr/bin/env node
// gcp-terraform-kit/scripts/state-mcp.mjs
import fs from "node:fs";
import path from "node:path";
import readline from "node:readline";
const cacheDirIndex = process.argv.indexOf("--cache-dir");
const cacheDir =
cacheDirIndex !== -1 && process.argv[cacheDirIndex + 1]
? process.argv[cacheDirIndex + 1]
: process.env.PLUGIN_DATA || "/tmp";
function inspectWorkspace(targetDir) {
const resolved = path.resolve(targetDir || ".");
if (!fs.existsSync(resolved)) {
return { error: `Directory not found: ${resolved}` };
}
const entries = fs.readdirSync(resolved);
const tfFiles = entries.filter((f) => f.endsWith(".tf") || f.endsWith(".tfvars"));
const hasLockFile = entries.includes(".terraform.lock.hcl");
fs.mkdirSync(cacheDir, { recursive: true });
const result = {
directory: resolved,
tfFilesCount: tfFiles.length,
tfFiles,
hasLockFile,
cacheDirUsed: cacheDir,
checkedAt: new Date().toISOString()
};
fs.writeFileSync(
path.join(cacheDir, "last-inspection.json"),
JSON.stringify(result, null, 2)
);
return result;
}
const rl = readline.createInterface({ input: process.stdin, terminal: false });
rl.on("line", (line) => {
if (!line.trim()) return;
let msg;
try {
msg = JSON.parse(line);
} catch {
return;
}
if (msg.method === "initialize") {
process.stdout.write(
JSON.stringify({
jsonrpc: "2.0",
id: msg.id,
result: {
protocolVersion: "2024-11-05",
capabilities: { tools: {} },
serverInfo: { name: "state-inspector", version: "1.0.0" }
}
}) + "\n"
);
} else if (msg.method === "tools/list") {
process.stdout.write(
JSON.stringify({
jsonrpc: "2.0",
id: msg.id,
result: {
tools: [
{
name: "inspect_terraform_workspace",
description:
"Lists .tf/.tfvars files, checks for .terraform.lock.hcl, and caches metadata in PLUGIN_DATA.",
inputSchema: {
type: "object",
properties: {
directory: {
type: "string",
description: "Path to the Terraform module directory."
}
},
required: ["directory"]
}
}
]
}
}) + "\n"
);
} else if (msg.method === "tools/call") {
const dir = msg.params?.arguments?.directory || ".";
const report = inspectWorkspace(dir);
process.stdout.write(
JSON.stringify({
jsonrpc: "2.0",
id: msg.id,
result: {
content: [{ type: "text", text: JSON.stringify(report, null, 2) }]
}
}) + "\n"
);
} else if (msg.id !== undefined) {
process.stdout.write(
JSON.stringify({ jsonrpc: "2.0", id: msg.id, result: {} }) + "\n"
);
}
});
> Tip — Prompting Users for API Keys (variables): If your plugin connects to a remote MCP endpoint that requires an API token, declare a variables array inside that server entry in mcp_config.json:
> ```json
> "variables": [
> {
> "name": "TFE_TOKEN",
> "title": "Terraform Registry API Token",
> "description": "Team or user API token with read access to private modules.",
> "link": "app.terraform.io"
> }
> ]
> ```
> When a server declares variables, agy plugin install stages the plugin in a disabled state until the user opens /plugin (or Customizations > Installed in the App) to fill in the token dialog.
---
Step 5: Define a Specialist Subagent (agents/iam-reviewer.md)
Custom agents live in agents/<agent-name>.md. By setting both subagent: true and mainAgent: true, this specialist can either be spawned automatically in the background by the primary agent or selected directly by the developer via agy --agent iam-reviewer (or /agents inside the TUI).
Notice how we omit edit_file, write_file, and run_command from the tools list to guarantee the reviewer operates strictly in read-only mode:
---
name: iam-reviewer
description: >-
Specialist auditor for Google Cloud IAM bindings, Service Accounts, and
Workload Identity Federation in Terraform. Delegate to this agent whenever
reviewing or modifying IAM permissions.
model: pro
mainAgent: true
subagent: true
commandExecutionPolicy: sandbox
tools:
- view_file
- grep_search
- find_by_name
- list_dir
- send_message
inheritMcp: true
---
You are a Principal Google Cloud Security Architect specializing in IAM and Terraform.
Your job is to inspect `.tf` files for privilege escalation paths, primitive roles (`roles/owner`, `roles/editor`), static keys (`google_service_account_key`), and authoritative bindings (`google_project_iam_policy`) that risk production outages.
For every issue found, return:
1. File path and line range.
2. The blast radius of the current configuration.
3. The exact drop-in HCL replacement using `google_project_iam_member` or Workload Identity.
---
Step 6: Enforce Deterministic Gates with Lifecycle Hooks (hooks.json)
Rules and system prompts guide the model's reasoning, but safety-critical boundaries belong in Lifecycle Hooks. Hooks execute synchronous shell commands when lifecycle events fire.
In a plugin, hooks.json must sit at <plugin-root>/hooks.json. Hook commands always execute with their working directory (cwd) set to <plugin-root>, so you should reference bundled scripts using relative paths (./scripts/block-apply.mjs) rather than ${PLUGIN_ROOT}.
// gcp-terraform-kit/hooks.json
{
"terraform-safety-gate": {
"enabled": true,
"PreToolUse": [
{
"matcher": "run_command",
"hooks": [
{
"type": "command",
"command": "node ./scripts/block-apply.mjs",
"timeout": 5
}
]
}
]
}
}
Now let us write scripts/block-apply.mjs. Hook scripts read the event payload as JSON from stdin (using camelCase keys like toolCall.args.CommandLine) and write a JSON decision object to stdout with exit code 0:
#!/usr/bin/env node
// gcp-terraform-kit/scripts/block-apply.mjs
import fs from "node:fs";
const rawInput = fs.readFileSync(0, "utf-8");
const payload = JSON.parse(rawInput || "{}");
const cmd = payload?.toolCall?.args?.CommandLine || "";
const forbiddenPattern = /\bterraform\s+(apply|destroy|state\s+rm|force-unlock)\b/i;
if (forbiddenPattern.test(cmd)) {
process.stdout.write(
JSON.stringify({
decision: "deny",
reason:
"Blocked by gcp-terraform-kit: state-mutating Terraform commands (apply, destroy, state rm, force-unlock) must run via CI/CD pipelines, not directly from the agent."
})
);
process.exit(0);
}
process.stdout.write(JSON.stringify({ decision: "allow" }));
Add a concise README.md at the plugin root for the Marketplace detail view, and mark both scripts in scripts/ as executable (0755) so Marketplace bundles preserve execution permissions:
cat << 'EOF' > gcp-terraform-kit/README.md
# GCP Terraform Kit
Google Cloud Terraform guardrails, module audit workflows, IAM reviewer subagent, and apply protection hooks for Google Antigravity.
EOF
chmod 0755 gcp-terraform-kit/scripts/*.mjs
---
4. Validating, Testing, and Installing Your Plugin
Before installing the plugin into Antigravity, we can smoke-test our PreToolUse hook and our state-inspector MCP server directly from the terminal:
# Test 1: Verify the hook blocks `terraform apply`
node ./gcp-terraform-kit/scripts/block-apply.mjs <<< '{"toolCall":{"name":"run_command","args":{"CommandLine":"terraform apply -auto-approve"}}}'
# Output: {"decision":"deny","reason":"Blocked by gcp-terraform-kit: state-mutating Terraform commands..."}
# Test 2: Verify the hook allows `terraform validate`
node ./gcp-terraform-kit/scripts/block-apply.mjs <<< '{"toolCall":{"name":"run_command","args":{"CommandLine":"terraform validate"}}}'
# Output: {"decision":"allow"}
# Test 3: Verify the MCP server responds to `initialize` and `tools/list`
printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}\n{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}\n' \
| node ./gcp-terraform-kit/scripts/state-mcp.mjs --cache-dir /tmp/tf-kit-cache
Now run the built-in validator from the Antigravity CLI (agy). Passing --strict treats warnings as errors and checks your plugin.json schema, skill directory naming, rule glob syntax, hook paths, and logo dimensions:
![Testing Locally with agy plugin validate --strict]
!Image description
# 1. Run strict validation against your plugin folder
agy plugin validate --strict ./gcp-terraform-kit
# Validating gcp-terraform-kit (.../gcp-terraform-kit/plugin.json)
# skills 1 · agents 1 · rules 2 · hooks 1 · mcpServers 1 · sidecars 0
# ✔ Validation passed
# 2. Install the plugin into ~/.gemini/config/plugins/gcp-terraform-kit
agy plugin install ./gcp-terraform-kit
# [ok] gcp-terraform-kit
# ✔ skills : 1 installed (audit-module)
# ✔ agents : 1 installed (iam-reviewer)
# ✔ rules : 2 installed (AGENTS, hcl-style)
# ✔ mcpServers : 1 installed (state-inspector)
# ✔ hooks : 1 installed (terraform-safety-gate)
# 3. Verify the plugin is enabled and its subagent is registered
agy plugin list
agy agents
Four Common Pitfalls Caught by --strict
If agy plugin validate --strict reports an error, check these four frequent mistakes first:
1. YAML Array in Rule globs: Writing globs: ["*.tf", "*.tfvars"] instead of a comma-separated string globs: "**/*.tf, **/*.tfvars".
2. Using ${PLUGIN_ROOT} in hooks.json: ${PLUGIN_ROOT} is expanded in mcp_config.json, not in hooks.json. Because hooks already run with cwd set to <plugin-root>, always use relative paths like node ./scripts/block-apply.mjs.
3. Skill Directory Mismatch: Naming the directory skills/tf-audit/ while setting name: audit-module inside SKILL.md. Both must match (skills/audit-module/SKILL.md).
4. Root-Level variables in mcp_config.json: Placing the variables array at the root of mcp_config.json instead of nesting it inside the specific server (mcpServers.<server-name>.variables).
---
5. Distributing Your Plugin to Your Team
Once your plugin passes strict validation, you have three ways to distribute it depending on your team's workflow:
Option A: Commit to a Repository (.agents/plugins/)
For repository-specific plugins that should activate automatically when any developer clones the repo, place the folder under .agents/plugins/gcp-terraform-kit/.
If your organization keeps shared plugins in a central monorepo directory (for example, platform/agent-plugins/), register that directory in .agents/plugins.json:
// .agents/plugins.json
{
"entries": [
{
"path": "platform/agent-plugins",
"exclude": ["experimental-.*"]
}
]
}
Option B: Install Directly from GitHub
Push gcp-terraform-kit to a GitHub repository. Any engineer on your team can install it across all their workspaces with a single command:
agy plugin install https://github.com/your-org/gcp-terraform-kit
Option C: Publish to the Antigravity Marketplace
If you are building an integration for an open-source tool, cloud platform, or developer product, add a README.md at the plugin root documenting prerequisites and example workflows, verify that agy plugin validate --strict exits with 0 errors and 0 warnings, and submit your repository through the official Google Antigravity Plugins documentation.
---
Wrapping Up
Moving from scattered prompt files to a structured Antigravity plugin takes less than fifteen minutes, and it changes how your team maintains AI workflows:
- Skills give engineers repeatable slash-command runbooks (/audit-module) without bloating the context window at startup.
- MCP servers wire up live schemas and internal APIs using portable ${PLUGIN_ROOT} and ${PLUGIN_DATA} variables.
- Subagents isolate heavy security and architecture reviews into dedicated read-only personas.
- Rules and Lifecycle Hooks pair context-aware guidance with deterministic PreToolUse safety gates.
Pick one workflow your team repeats every week—whether that is auditing Terraform modules, reviewing Firestore security rules, or verifying Kubernetes manifests—package it into a plugin.json bundle, and run agy plugin validate --strict.
1 reaction
Give Claude Code and Cursor a shared, auto-updating map of your codebase
AI coding assistants are great at writing code and surprisingly bad at knowing where things are. Ask "what breaks if I change this function?" on a large repository and the assistant starts opening files one by one, burning tokens and context, and often still misses a caller three modules away.
Graphify fixes that by turning a codebase into a knowledge graph: functions, classes, modules and the edges between them, parsed locally with tree-sitter. Your assistant queries the graph through MCP instead of re-reading the code. With over 130k GitHub stars, it clearly hit a nerve.
This post shows how to run Graphify as one shared MCP server that every assistant on your team can use, kept up to date automatically. Everything here is open source.
Local graph vs. shared server
The default Graphify workflow is local: you run it in your repo, it writes graphify-out/graph.json, and your assistant reads it. If you work alone on one machine, that's perfect. Committing graphify-out/ and installing Graphify's git hook even keeps several of your own machines in sync.
It gets clumsier when:
- several people (or several agents) need the same graph,
- you don't want everyone to install and run Graphify locally,
- you want the graph to follow main without anyone thinking about it.
Graphify already ships an HTTP transport for its MCP server, so the missing piece is just the plumbing: clone, build, serve, refresh, secure. That's what graphify-railway does.
How it works
your Git repo ──clone/pull──▶ graphify extract --code-only ──▶ graph.json (volume)
│ reloaded on change
Claude Code / Cursor ──HTTPS + API key──▶ /mcp ◀── graphify.serve --transport http
1. Clone. On start, the service clones REPO_URL. Private repos work with a read-only token.
2. Build. It runs graphify extract --code-only: pure local AST parsing, no LLM API key. On a small repo (about 100 nodes) it takes under a second.
3. Serve. It starts graphify.serve over MCP Streamable HTTP on /mcp. Every request must carry the API key.
4. Refresh. Every hour (configurable), it pulls the repo. If the commit changed, it rebuilds and swaps graph.json atomically. Graphify's server checks the file on each request and reloads it, so connected clients keep their session: no restart, no downtime.
A few details I cared about:
- The git token never touches the disk. It's passed to each git command as an HTTP header, not written to .git/config, and it's redacted from logs.
- The API key isn't on the command line. It goes through the environment, so it doesn't show up in ps.
- The last good graph survives. It lives on a persistent volume: if the repo is briefly unreachable at startup, the previous graph is served; if a refresh fails, the old graph stays online.
- Graphify is pinned to a tested release in the Dockerfile. Graphify ships very often, and you don't want a surprise upgrade on a team server.
The whole wrapper is one Python file and a Dockerfile.
Deploy it
One click on Railway
Deploy on Railway. You get the service and a volume. Fill in one variable, REPO_URL (for example https://github.com/your-org/your-repo); the API key is generated for you. For a private repo, also set GIT_TOKEN. On GitHub, use a fine-grained token limited to that repository with Contents: Read-only.
Anywhere you can run a container
docker build -t graphify-server https://github.com/jean82290/graphify-railway.git
docker run -d -p 8080:8080 \
-v graphify-data:/data \
-e REPO_URL=https://github.com/your-org/your-repo \
-e GRAPHIFY_API_KEY="$(openssl rand -hex 24)" \
graphify-server
Put it behind HTTPS before sharing it with anyone.
| Variable | Required | Default | What it does |
|---|---|---|---|
| REPO_URL | yes | | HTTPS URL of the repository to map |
| GRAPHIFY_API_KEY | yes | | Key every MCP client must send |
| GIT_TOKEN | private repos | | Read-only access token |
| REPO_BRANCH | no | default branch | Branch to map |
| REFRESH_MINUTES | no | 60 | How often to pull and rebuild (0 disables) |
Connect your assistant
Claude Code:
claude mcp add --transport http graphify https://YOUR-SERVICE.up.railway.app/mcp \
--header "Authorization: Bearer YOUR_GRAPHIFY_API_KEY"
Cursor (.cursor/mcp.json):
{
"mcpServers": {
"graphify": {
"url": "https://YOUR-SERVICE.up.railway.app/mcp",
"headers": { "Authorization": "Bearer YOUR_GRAPHIFY_API_KEY" }
}
}
}
Any MCP client that supports Streamable HTTP with custom headers works the same way.
What your assistant can do with it
Graphify exposes tools such as query_graph, get_node, get_neighbors, get_community, god_nodes, graph_stats and shortest_path, plus PR-oriented tools (list_prs, get_pr_impact, triage_prs).
In practice, this means questions like these get answered from the graph instead of a file-by-file crawl:
- "What calls buildEvent, directly or indirectly?"
- "What's the shortest path between the HTTP handler and the database layer?"
- "Which parts of the code are the most connected, and therefore the riskiest to change?"
When I tested it on a small Node.js project, a query_graph call starting from one function returned its 80 related nodes and 152 edges in a single response. That's the context an assistant would otherwise have to rebuild by opening a dozen files.
Security notes
- The server is reachable on the internet, and the API key is what keeps your code map private. Rotate it by changing GRAPHIFY_API_KEY and redeploying.
- The graph contains names and relationships from your code, not the full source, but treat it as confidential if your code is.
- Use a token with read access to a single repository.
Limitations
- Code only, for now. Graphify can also extract docs, PDFs and images, but that pass needs an LLM backend and an API key. The wrapper sticks to the free, offline AST mode.
- One repository per instance. If you need several, run several instances (or tell me you'd like multi-repo support).
- Hourly refresh by default. Lower REFRESH_MINUTES if you need fresher graphs.
Links
- Source (MIT): jean82290/graphify-railway
- One-click deploy: Railway template
- Graphify itself (Apache-2.0): Graphify-Labs/graphify
Disclosure: this is an unofficial wrapper, not affiliated with Graphify Labs. If you deploy through the Railway template, I receive a share of Railway's hosting fees. The Docker route above costs me nothing and works just as well.
Questions, ideas or bugs: open an issue on GitHub or leave a comment below. I'm especially curious whether multi-repo support would be useful to you.
Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins
How the FiftyOne plugin framework turns your MCAP and LeRobot curation checks into buttons, panels, and background jobs that run on every episode, and why that works across a whole dataset rather than one recording.
You have 400 teleoperation episodes and a policy to train by Friday.
Some of those episodes are bad. The operator hesitated for eight seconds before the grasp. The wrist camera froze halfway through. The gripper opened and closed three times because the first grasp slipped. The task string says "pick up" and nothing else.
A policy will learn from all of it.
You could open each episode in a log viewer and scrub through it. At two minutes per episode, that's more than 13 hours, and you'll stop paying attention long before the end. What you want is a ranked list, worst first, with every flag one click away from the moment it happened.
No viewer ships that list, because the checks depend on your robot. A dual-arm rig with multi-joint hands needs different smoothness metrics than a wheeled rover with an inertial measurement unit (IMU) and GPS. The tool that curates your data has to be one you can extend in Python, on your schema, across the whole dataset.
That's what the FiftyOne plugin framework is for.
Key takeaways
- A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job.
- FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code.
- FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution.
- Most viewer extensions run in the browser against one recording, and labeling-platform hooks act on one workflow task. FiftyOne plugins run server-side Python with custom UI across a whole dataset.
- Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings.
Why physical AI data needs custom tooling
Image datasets are forgiving. Every sample is a picture, every label is a box or a mask, and the same quality checks work on most of them.
Robot data has no such standard shape. One MCAP file from an autonomous vehicle might hold six cameras, two LIDARs, an IMU, GPS fixes, and a /tf tree. One LeRobot episode from a tabletop arm holds two camera streams, a 14-dimensional observation.state, a matching action array, and a task string. An egocentric dataset adds audio and hand poses.
The questions you need answered are specific to that shape:
- Is the motion in this episode smooth, or full of corrections?
- Did any sensor drop frames, drift its clock, or clip its values?
- Does the action track the state, or is there lag between video and action?
- Is the gripper behavior consistent with the other episodes of the same task?
- Is the language instruction usable for fine-tuning a vision-language-action (VLA) model?
A generic viewer can't answer those. It can only show you the data and wait for you to notice.
FiftyOne gives you the dataset layer: episodes as samples, fields you can filter and sort on, views, tags, and an App with a multimodal viewer that plays MCAP and LeRobot streams in lockstep. Plugins give you the part that's specific to your robot.
!Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation.
Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation.
What is a FiftyOne plugin?
A plugin is a folder FiftyOne finds. If the folder contains a fiftyone.yml, FiftyOne loads it at runtime and adds whatever it declares to the App, the SDK, and the CLI.
Plugins add two kinds of things:
- Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator.
- Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel.
Under the hood, a panel is a special kind of operator, so everything below about operators applies to panels too.
| | Notebook script | FiftyOne plugin operator |
| --- | --- | --- |
| Who can run it | You, in your kernel | Anyone with the App open, no code |
| Inputs | Variables you edit by hand | A typed form, validated before anything runs |
| What it acts on | The dataset you loaded | The view or selection the user has open |
| Long jobs | Your kernel is busy until it finishes | Delegated to a worker, so the App stays free |
| Reuse | Copy and paste | One URI: buttons, panels, pipelines, notebooks, agents |
What changes when you promote a notebook script to a FiftyOne plugin operator.
Good to know. Why not keep it in a notebook? Start in a notebook. Promote the code to an operator when someone else needs to run it, or when you catch yourself running it a second time. The figure below shows what changes.
!Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse.
Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse.
The row that matters most for robot data is "what it acts on". A script scores whatever dataset you loaded. An operator scores the view the user has open right now, so the same code works on one task, one robot, one week of collection, or the episodes someone tagged for review.
The anatomy of a FiftyOne plugin
Here's the folder for a plugin with a Python backend and a custom React panel:
!FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App.
FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App.
- fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs.
- __init__.py holds your classes and a register(p) function.
- requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends.
- Everything below the dashed line is optional JavaScript for custom UI.
A Python-only plugin is two files. Teal in every figure in this post means the FiftyOne server, which runs Python. Orange means the App in your browser.
Keep it honest. Every operator name listed in fiftyone.yml must match the name in that operator's config. If you register a class but forget to list it, the operator doesn't show up, and you get no error. It's the most common "where did my operator go" bug.
A complete operator in 25 lines
This operator tags samples whose detections fall below a confidence threshold:
import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.operators.types as types
from fiftyone import ViewField as F
class TagLowConf(foo.Operator):
@property
def config(self):
return foo.OperatorConfig(name="tag_low_conf", label="Tag low-confidence")
def resolve_placement(self, ctx):
return types.Placement(
types.Places.SAMPLES_GRID_SECONDARY_ACTIONS,
types.Button(label="Tag low-conf"),
)
def resolve_input(self, ctx):
inputs = types.Object()
fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections))
inputs.enum("field", fields, required=True, label="Label field")
inputs.float("thresh", default=0.3, label="Confidence threshold")
return types.Property(inputs)
def execute(self, ctx):
field, thresh = ctx.params["field"], ctx.params["thresh"]
view = ctx.target_view().filter_labels(field, F("confidence") < thresh)
view.tag_samples("low_conf")
ctx.ops.notify(f"Tagged {len(view)} samples")
def register(p):
p.register(TagLowConf)
And here's what the user sees:
!TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App.
TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App.
Four methods, four jobs:
1. config names the operator. name is its ID, and label becomes the form's heading.
1. resolve_placement puts a "Tag low-conf" button in the grid's action bar.
1. resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines.
1. execute does the work on the target view and sends a notification.
You wrote no UI code. You described the inputs and the App built the interface. Swap the body of execute for spectral arc length (SPARC) smoothness over an /odom channel, and you have the skeleton of an MCAP quality scorer.
How an operator call works
Every operator call is two round trips between the App and the FiftyOne server:
!Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations.
Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations.
1. Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel.
1. Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them.
Your code runs only on the server. The browser draws what the server tells it to. That's why an operator can decode protobuf MCAP messages with mcap-protobuf-support, run SciPy's Welch power spectral density (PSD) on an IMU stream, or call a model on a GPU, without shipping any of that to the browser.
When something breaks, ask which arrow failed. A wrong form means resolve_input. A fine form that does nothing means execute. An execute that ran but didn't change the App means the UI operations you returned.
Long jobs run in the background with delegated execution
Scoring 400 episodes is not a click-and-wait job. Decoding video alone costs lerobot_data_curation about 0.4 seconds per camera per episode.
!Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them.
Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them.
Set allow_delegated_execution=True in the operator's config, and the user can choose. Immediate execution runs inside the server while the App waits, with a live progress bar. Delegated execution writes the job to a queue in MongoDB, and a separate worker runs the same execute method. You start a worker with fiftyone delegated launch and track jobs with fiftyone delegated list.
Both physical AI plugins below default to delegated execution, because a real corpus takes minutes.
Keep it honest. If no worker is running, delegated jobs sit in the queue forever. And a delegated job has no browser attached, so write results to the dataset rather than trying to change the user's view.
Panels: UI that stays open
A curation tool needs more than a form. It needs a place to see the ranking, click into an episode, and tag it. That's a panel.
!FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields.
FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields.
A panel's on_load runs when it opens, render returns a layout, and events call your Python handlers with a fresh context. The event that matters most for curation is on_change_view: when the user filters the grid, your panel re-computes for the new view.
Here's a complete panel that shows class counts and follows the grid:
class ClassCounts(foo.Panel):
@property
def config(self):
return foo.PanelConfig(name="class_counts", label="Class counts")
def on_load(self, ctx):
self.refresh(ctx)
def on_change_view(self, ctx):
self.refresh(ctx)
def refresh(self, ctx):
counts = ctx.view.count_values("ground_truth.detections.label")
ctx.panel.set_state("n", len(counts))
ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}])
def render(self, ctx):
panel = types.Object()
panel.md(f"**{ctx.panel.get_state('n')} classes**")
panel.plot("counts")
return types.Property(panel)
Small values go in panel state, which your Python handlers can read. Large payloads like chart data go through set_data, which is write-only from Python. Settings you want to survive a reload go in the execution store, ctx.store. Results that belong to the data go in dataset fields, where you can filter and sort on them like any other field.
When the built-in widgets run out, you have two options. A hybrid panel keeps state and compute in Python and renders a React component you write. A JS panel puts React in charge of the UI and calls your Python operators whenever it needs the server. Both plugins below are JS panels backed by Python operators, which suits dense, interactive dashboards. If you build a React UI, VOODO, the component library the FiftyOne App itself uses, makes it look native.
!Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build.
Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build.
Operators compose
Every operator has a URI, like @voxel51/brain/compute_similarity, and a typed input schema. Anything that can call an operator can call yours.
!One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates.
One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates.
That includes the FiftyOne Model Context Protocol (MCP) server, which exposes list_operators, get_operator_schema, and execute_operator to AI agents. An agent can find your quality scorer, read its inputs, run it on the episodes tagged review, and report back which ones failed sensor health. The schema validates the agent's JSON the same way it validates a person filling in the form.
For a physical AI team, that means the curation logic you write once runs from a button during review, from a notebook in your training pipeline, from a background worker overnight, and from an agent.
How FiftyOne plugins differ from viewer extensions
Most robotics viewers have extension points, and they're usually scoped to the recording you have open: a custom panel, a message converter, or a new file loader that runs in the browser or desktop app. Labeling platforms add hooks that act on one task in a workflow stage.
FiftyOne plugins run on the server, in Python, against any view of a dataset. A single plugin can add a form, run a job across every episode in the background, render a custom panel with the results, and expose all of it to agents through the MCP server, without leaving the language your robotics stack is written in.
For live debugging of a single recording, a dedicated log viewer is the right tool, and FiftyOne doesn't try to replace it. FiftyOne reads .rrd files and MCAP files, so a common setup is to debug individual logs in a viewer and curate the dataset in FiftyOne.
Two FiftyOne plugins for MCAP and LeRobot curation
Both plugins below were built using only the pieces covered above. Both follow the same rule, stated in their READMEs: the scores order your review queue, and a person makes the call. A perfectly smooth demonstration can still show the wrong task.
LeRobot Data Curation
lerobot_data_curation ranks the episodes of a LeRobot v3 dataset worst-first, so you know which to look at before training a policy or fine-tuning a VLA.
It scores each episode on motion smoothness, time efficiency, action-state tracking, gripper behavior, consistency, integrity, and language, and optionally on camera quality: blur, exposure, clipping, frozen feeds, and video-action lag. It has no model-based metrics, so there's no model to download.
How it maps to the framework:
- A typed form with tabs. The compute operator's form has Data, Metrics, Camera, and Normalization tabs. You pick the observation.state and action arrays, and assign arm and gripper joints by name. Nothing is guessed from the data, and your picks are remembered for the next run.
- Delegated by default. The operator sets allow_delegated_execution=True and default_choice_to_delegated=True, so scoring runs in the background. Camera decoding is the slow part.
- A JS panel backed by Python operators. The LeRobot Curation panel is a React component with Overview, Motion & Action, Integrity & Coverage, Vision and Language tabs. Click a histogram bar to filter the grid to those episodes. Click a row to open an inspector with joint traces, the speed profile, the gripper timeline, and frames picked around the flagged spans.
- It follows the view. The panel refreshes when the view changes, so filtering the grid re-ranks it.
- Tags on the timeline. Flagged spans such as idle stretches, the longest pause, and acceleration spikes are written as temporal tags on each episode so that you can scrub straight to them in the multimodal viewer. Footer buttons apply review, exclude-candidate, and relabel sample tags.
import fiftyone as fo
dataset = fo.Dataset.from_dir(
dataset_dir="/data/lerobot/my_dataset",
dataset_type=fo.types.LeRobotDataset,
)
session = fo.launch_app(dataset)
# then run "LeRobot curation: compute quality" from the operator browser
Keep it honest. Scores are relative to the batch you scored. Per-task normalization needs about 20 episodes per task, and the default thresholds were calibrated on a 102-episode development set. Check them on your own data. Cloud-hosted LeRobot sources aren't supported yet.
Demo Quality Scorer
demo_quality_scorer does the same job for multimodal MCAP datasets: teleoperations, autonomous vehicles, drones, rovers, and egocentric capture.
!Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer.
Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer.
It computes three metric families and ranks episodes worst-first:
1. Motion smoothness: SPARC, log dimensionless jerk (LDLJ), jerk RMS, and a low-to-high frequency power ratio, computed over windows of a speed profile you choose from position or velocity signals such as /odom -> twist_linear.
1. Sensor health: dropout, rate stability, clock drift, clipping, and cross-channel desync, all from message timestamps and raw values, so it works on any channel the plugin can decode.
1. Outliers: models that flag episodes whose motion and health features don't look like the rest of the batch.
How it maps to the framework:
- A dynamic form. The form inspects the first sample's channels. If no telemetry channel carries a numeric signal, the motion family switches itself off and says why. If the server lacks a decoder, the form names the package to install.
- Immediate or delegated, your choice. Run immediately with an in-app progress bar, or schedule it in the background (default).
- A JS panel. The Episode Quality panel is a React component that has Motion, Health, and Outliers tabs. Click a histogram bar to filter the grid. Click a table row to open that episode in the multimodal viewer, with a toast naming the timecode of its worst flagged interval.
- Results as fields. Scores land in quality.* fields such as quality.sparc and quality.ldlj, so you can sort and filter on them outside the panel too.
pip install "fiftyone[multimodal]>=1.22.0" mcap mcap-protobuf-support \
mcap-ros1-support mcap-ros2-support numpy scipy scikit-learn
fiftyone plugins download https://github.com/harpreetsahota204/demo_quality_scorer
Keep it honest. Structural errors, such as the wrong action at a key moment, are invisible to every metric here. Re-running over a different set of episodes re-fits normalization, so expect scores to move.
Build your own in six steps
Your robot has checks neither plugin covers. Here's the loop for writing them.
!Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write.
Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write.
Step 1: Scaffold the plugin
Run fiftyone plugins create or make a folder in your plugins directory with a fiftyone.yml:
name: "@your-org/gripper_checks"
description: Flags episodes with repeated grasp attempts
operators:
- score_grasps
Step 2: Write the operator
One class per operator or panel, like the examples above. Keep the scoring logic in a plain Python function so a notebook, a test, and the operator can all call it.
Step 3: Register it
Call p.register(ScoreGrasps) inside register(p), and make sure the name matches fiftyone.yml.
Step 4: Run the App in debug mode
export VFF_MULTIMODAL=1
fiftyone app debug my_lerobot_dataset
fiftyone app debug prints server logs and tracebacks to your terminal. VFF_MULTIMODAL=1 turns on the multimodal viewer in the App; set it before fiftyone is imported in every process that touches a multimodal dataset.
Step 5: Iterate
Edit, refresh the App, repeat. Python changes are picked up on the next request without a server restart. If you have a React panel, keep npm run dev rebuilding it.
Step 6: Ship it
Push to GitHub. Anyone installs it with:
fiftyone plugins download https://github.com/your-org/gripper_checks
fiftyone plugins requirements @your-org/gripper_checks --install
Good to know. How do I test an operator without the App? Operators are callable from Python with foo.execute_operator, so you can test them headless in pytest.
Try it yourself
- Plugin developer guide
- FiftyOne multimodal user guide for MCAP and LeRobot
- voxel51/fiftyone-plugins, the official plugins and panel examples
- lerobot_data_curation
- demo_quality_scorer
- FiftyOne MCP server
- VOODO, the React design system for plugin panels
Frequently Asked Questions
How do I curate a LeRobot dataset before training a policy?
Import it with fo.Dataset.from_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset), where each sample is one episode. Then install lerobot_data_curation and run its compute operator to rank episodes worst-first on motion, gripper, camera, and language metrics. Review the top of the list and tag episodes as exclude-candidate or relabel.
Can FiftyOne open MCAP files?
Yes. FiftyOne treats .mcap, .bag, and .rrd files as media_type="multimodal" samples, one episode per file, and plays their streams in a synchronized multimodal viewer. Set VFF_MULTIMODAL=1 before importing fiftyone to turn the viewer on.
How do I find bad demonstrations in robot teleoperation data?
Score every episode on motion smoothness (SPARC, LDLJ, jerk) and sensor health (dropout, rate stability, clock drift), then sort worst-first. demo_quality_scorer does this for MCAP datasets. Treat the scores as a review queue, because a smooth demonstration can still show the wrong task.
Do FiftyOne plugins need JavaScript?
No. A Python-only plugin is fiftyone.yml plus __init__.py, and the App builds forms and panels from Python declarations. Add React only when you need custom interactivity, such as the dashboards in lerobot_data_curation and demo_quality_scorer.
Where does FiftyOne plugin code run?
It runs on the FiftyOne server, in Python, or on a background worker for delegated operations. It never runs in the browser, so it can use any Python library, a GPU, or secrets without exposing them.
I built a free validated n8n template library with 54 workflows (CI-tested, documented) — feedback welcome
# Release v0.2.0 — OpenAI Ecosystem Focus + First Validated Workflows
## What's new
### ✅ First ready-status workflows
Two workflows are now fully validated with complete setup documentation, sample payloads, SQL scripts, and cost tables:
- [Translate Form Submissions](AI_ML/translate_form_submissions.json) — GPT-4o + text-embedding-3-small + Supabase pgvector
Translates multilingual webhook form payloads and logs results to Google Sheets.
- [Daily Content Ideas](AI_ML/daily_content_ideas.json) — GPT-4o + Cohere embeddings + Supabase deduplication
Generates and deduplicates daily content ideas, posts to Slack.
### 📋 OpenAI workflow index
The root README now has a dedicated section listing all 10 OpenAI-powered workflows with their specific models (gpt-4o, text-embedding-3-small, whisper-1).
### 🔧 Stronger CI
- JSON schema validation added to scripts/validate_workflows.py via optional jsonschema dependency
- scripts/requirements.txt makes CI dependencies reproducible
- CI installs dependencies before running validator
### 📖 Richer docs for validated workflows
Validated workflow docs now include:
- Full node inventory table
- Sample input JSON payloads
- Sample output format
- Step-by-step credential setup
- Supabase pgvector SQL
- Per-provider cost estimates
## Install / use
```bash
# Clone and validate
git clone your-org/n8n-free-templates
cd n8n-free-templates
pip install -r scripts/requirements.txt
python scripts/validate_workflows.py
```
Import any .json file into n8n via Settings → Import from File.
## What's next (v0.3.0)
- Validate remaining AI/ML workflows
- Add category READMEs with compatibility notes
- Add error handling and retry guidance to workflow docs
1 point · 2 comments
I built a free validated n8n template library with 54 workflows (CI-tested, documented) — feedback welcome
# Release v0.2.0 — OpenAI Ecosystem Focus + First Validated Workflows
## What's new
### ✅ First ready-status workflows
Two workflows are now fully validated with complete setup documentation, sample payloads, SQL scripts, and cost tables:
- [Translate Form Submissions](AI_ML/translate_form_submissions.json) — GPT-4o + text-embedding-3-small + Supabase pgvector
Translates multilingual webhook form payloads and logs results to Google Sheets.
- [Daily Content Ideas](AI_ML/daily_content_ideas.json) — GPT-4o + Cohere embeddings + Supabase deduplication
Generates and deduplicates daily content ideas, posts to Slack.
### 📋 OpenAI workflow index
The root README now has a dedicated section listing all 10 OpenAI-powered workflows with their specific models (gpt-4o, text-embedding-3-small, whisper-1).
### 🔧 Stronger CI
- JSON schema validation added to scripts/validate_workflows.py via optional jsonschema dependency
- scripts/requirements.txt makes CI dependencies reproducible
- CI installs dependencies before running validator
### 📖 Richer docs for validated workflows
Validated workflow docs now include:
- Full node inventory table
- Sample input JSON payloads
- Sample output format
- Step-by-step credential setup
- Supabase pgvector SQL
- Per-provider cost estimates
## Install / use
```bash
# Clone and validate
git clone your-org/n8n-free-templates
cd n8n-free-templates
pip install -r scripts/requirements.txt
python scripts/validate_workflows.py
```
Import any .json file into n8n via Settings → Import from File.
## What's next (v0.3.0)
- Validate remaining AI/ML workflows
- Add category READMEs with compatibility notes
- Add error handling and retry guidance to workflow docs
2 points · 4 comments
Claude Took Anthropic From 2,500 to 30,000 Merges a Day. Then It Moved Off Git
- Book: AI Agents Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
---
In May 2026, Anthropic merged around 2,500 code changes a day. By September it was merging more than 30,000 a day, according to the company's engineering blog.
Anthropic names the cause itself. Claude's ability to write code, the post says, is "what put us in this situation in the first place." It also names where things gave way first: "The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI."
On October 13, Anthropic published what it did about it. A team of seven built Scion, an in-house version control system, plus a code review tool called Antique, and fully migrated the monorepo off git in five months. The post drew a busy Hacker News thread within hours.
What broke at 30,000 merges a day
The scale numbers in Anthropic's write-up explain why. "The history of our main branch alone is almost 100 GB, and a compressed working copy is over 25 GB." Anthropic runs "on the order of 25 million code checkouts per day for CI." Then the line that ends the argument: "Fetched naively, that would be over 600 PB of git traffic per day."
Anthropic's diagnosis is short: "Git assumes that clients work offline and need a complete copy of the repository." That design is why git is pleasant on a laptop. It is also why every CI job that wants one directory still has to negotiate with history it will never read. Anthropic had been absorbing that load with a fleet of git mirrors, which it now calls "our previous git mirroring infrastructure."
!A classic branching version-control graph drawn as a tree, overwhelmed by a dense swarm of small orange commit nodes pouring in from robotic hands, with a narrow pipe at the base straining to carry them
When agents write the code, the repository's plumbing becomes the slow part.
How Scion breaks git's assumptions
Scion starts from the opposite premise, which Anthropic states in four words: "Clients are always online." The rest of the design follows from that:
- The server owns history, blame, commits, merges and rebases. The client keeps a bounded local cache and the working files.
- Checkouts are sparse by default. Scion's protocol "is centered around sparse checkouts, which only contain the directories the client asks for." Some jobs skip the checkout entirely: "a CI job that needs the diff between a commit and its merge base, for example, can ask the server for it."
- Fetches go through tiers: the scn client talks to a per-node scion-daemon, which talks to regional scion-cache servers, which talk to scion-server instances backed by Spanner or cloud storage. Anthropic reports over 99% cache hits at each tier.
- Branches are mostly gone. Scion "discourages refs (pointers to commits, such as branches and tags)," except for the main ref and release branches, because shared global state is what stops scaling.
The results Anthropic reports: an infrastructure footprint "200x smaller than our previous git mirroring infrastructure," checkouts for CI and research jobs that take "seconds rather than minutes," and "a 25% CPU capacity win" for CI alone. These are Anthropic's own numbers on its own systems, and Anthropic hasn't released anything that would let outsiders reproduce them.
None of this is new as a category. Anthropic itself points to Google's Piper and Meta's Sapling as earlier server-centric systems. What is new is the reason: Anthropic built one because its own coding model outran its tooling.
The process changes matter more than the VCS
The part most teams can learn from has nothing to do with storage.
Start with the merge queue. "A proper merge queue, for example, doesn't make much sense at agent scale," Anthropic writes. It moved the safety net later instead: "we've invested in post-merge and pre-deploy automated testing."
Review changed too. "Our team members still review every change request for the system," but "they now examine the intent of the change as opposed to reading every line of code."
Anthropic does not claim the model got everything right. "Agents will sometimes violate core system constraints to satisfy a pattern." One adversarial review "was overly defensive about trusting another internal service," and Claude piled on extra validation to satisfy it.
The upside was large too. The team handed an early internal snapshot of Claude Mythos Preview the whole ingestion pipeline to rewrite, and the new version, per Anthropic, "consumed 20x less memory and was 4x faster." Every engineer who worked on the system would now rate Claude as better at writing code than themselves, the post adds.
!Abstract illustration of a request travelling down a tiered stack of cache layers, from a small client node through a middle cache ring to a large central server cylinder, with most requests stopping early at the upper tiers
Scion's fetch path: client, node daemon, regional cache, server. Most requests never reach the bottom.
Should you leave git? Anthropic says no
Anthropic is blunt about this. "Scion is by no means a suitable replacement for git for general use." The concepts "really only make sense for truly enormous monorepos." Anthropic has not announced a public release, and it describes its internal systems as highly bespoke.
What it asks for instead is better git infrastructure: "Git platforms can provide comprehensive server-side RPCs like Scion," and "A git server doesn't have to work the same way as a git client does."
Two of Scion's ideas already exist in stock git at a smaller scale. A partial clone skips file contents until you need them, and sparse checkout only materialises the directories you name. Both are flags on git clone. For a CI job that touches one service:
git clone --filter=blob:none --sparse \
https://github.com/your-org/your-repo.git
cd your-repo
git sparse-checkout set services/billing
It won't get you anywhere near Anthropic's numbers, but the instinct is the same: stop shipping the whole repository to a job that reads one folder.
My reading of the post: most teams will not hit 30,000 merges a day, but plenty will hit the moment when agents open changes faster than CI and review can absorb them. That is the moment to look at your merge queue and your review rules, long before you look at your VCS.
Several agents on one repo, at laptop scale
I build Hermes IDE, so weigh this section with that in mind. Hermes is not affiliated with Anthropic.
At Anthropic's volume, branches are shared global state worth avoiding. On one machine with three agents, the problem is smaller and different: two agents in one checkout overwrite each other. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own git worktree, with the task as the agent's first prompt. Claude Code, Codex, Antigravity CLI and other command-line agents each run in a real terminal.
The review side is closer to what Anthropic describes. The Review Desk (⌘G) shows an agent's changes by turn or by file, with risk flags, and sends a line comment back to the agent that wrote it. Reviewing one turn at a time makes it easier to judge the intent behind a change. When you are done, Land commits, opens a pull request or merges locally, and can be undone.
How many agent-written changes a day can your CI take before it becomes the queue? Tell me in the comments.
---
If this was useful
Hermes IDE is free, source-available (BSL 1.1), and runs on macOS, Windows and Linux. Download it and give two agents the same ticket on separate branches, or star it on GitHub if keeping agents apart in one repo matters to you. If you build agents yourself, the AI Agents Pocket Guide covers the patterns behind agents that write and review code, like the ones in Anthropic's post.
7 reactions
From Github to Container: Running a Node.js App (Wonderkid app) Locally and Packaging It with Docker.
Getting a project running on your machine and then packaging it in a container are two of the most common tasks in modern development and DevOps work. This guide takes a Node.js web application (wonderkid) from a fresh git clone to a running Docker container, with an explanation of why each step matters.
Prerequisites: Git, Node.js (v22 recommended) with npm, VS Code, and Docker Desktop.
#Part 1: Local Application Setup
Before containerizing anything, confirm the application works on your own machine. If it doesn't run locally, it won't run in a container either, and debugging is much easier outside Docker. Lets follow the following steps to confirm this:
##Step 1: Clone the repository
Open VS Code and create a dedicated workspace folder for your projects (in this case, I used the date I created the folder to name the folder ‘oct326’ ). Keeping projects in one place makes them easier to find and manage. Open the integrated terminal (Terminal → New Terminal, or Ctrl + `), navigate to that folder, and clone the repository. To do this you will need the repository url which you will get from the GitHub repository. :
git clone <repository-url>
git clone https://github.com/your-org/wonderkids.git
!git clone
This downloads the full project, including its history, into a new folder named after the repository.
##Step 2: Navigate into the project
Before navigating into the folder, type the Ls command to see that the folder exists
!ls
Then use the ‘cd’ command to navigate inside the project folder.
cd <folder-name>
!Cd
Every command from here on must run from the project root, the folder containing package.json. Running ls (macOS/Linux) or dir (Windows) should show package.json, src/ or app/, and similar files. If you don't see package.json, you're in the wrong folder.
##Step 3: Install dependencies
npm install
!npm install
npm reads package.json (and package-lock.json, if present) and downloads everything the project needs into a node_modules/ folder. This can take a minute or two. Some warnings about deprecated packages are normal, but red ERR! lines mean something failed, often a Node version mismatch, so check the project's README for the required version.
##Step 4: Run the app locally
npm run dev
!npm rundev
This starts the development server, which usually prints something like:Local: localhost
Open that address in your browser to confirm the application loads. Dev servers typically include hot reloading, so changes to your code appear instantly.
!Open
When you're finished, stop the server by pressing
Ctrl + C in the terminal.
Tip: If the port is already in use, another process is occupying it. Stop that process or run the app on a different port, depending on your framework's configuration.
#Part 2: Docker Application Containerization
Containerization packages your application together with its runtime and dependencies, so it behaves the same on your laptop, a teammate's machine, a CI pipeline, or a cloud host. These are the steps I took to contanerize wonderkid application
##Step 1: Check that Docker Desktop is running
Docker commands need the Docker engine running in the background. Launch Docker Desktop and wait until its status shows as running. Then verify from the terminal:
docker --version
docker info
If docker info returns an error like "Cannot connect to the Docker daemon," Docker Desktop isn't running yet.
##Step 2: Create the configuration files
From the project root, create the two files Docker needs:
touch Dockerfile
!touch
touch .dockerignore
!Docker ignore
Alternatively, you can create it manually without using the codes.
On Windows PowerShell, use New-Item Dockerfile and New-Item .dockerignore instead, or create the files from VS Code's Explorer panel.
Dockerfile is the recipe that tells Docker how to build your image.
.dockerignore lists files that should not be sent to Docker during the build.
##Step 3: Configure the .dockerignore
!Docker ignore config
Without a .dockerignore, Docker copies everything in your folder into the build context, including a large node_modules directory and your Git history. That slows builds and can leak sensitive files. Add:
node_modules
npm-debug.log
.next
dist
build
.git
.gitignore
.env
.env.*
Dockerfile
.dockerignore
README.md
.vscode
Excluding node_modules is especially important. The container installs its own dependencies for its own operating system, and copying yours in can cause conflicts.
##Step 4: Write the Dockerfile with a multi-stage build
!Dockerfile config
A multi-stage build uses several FROM stages in one Dockerfile. Build tools and development dependencies are needed to compile your app but not to run it. By building in one stage and copying only the results into a clean final stage, you get a smaller, faster, and more secure image.
Here is a solid starting template using Node 22 on Alpine Linux, a minimal base image:
Stage 1: Install dependencies ----------
FROM node:22-alpine AS deps
WORKDIR /app
Copy only dependency manifests first to leverage Docker layer caching
COPY package.json package-lock.json ./
RUN npm ci
Stage 2: Build the application ----------
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
Stage 3: Production runtime ----------
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV PORT=3000
Run as a non-root user for better security
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
Copy only what is needed to run the app
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/package-lock.json ./package-lock.json
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
Remove dev-only packages from the final image
RUN npm prune --omit=dev && chown -R appuser:appgroup /app
USER appuser
EXPOSE 3000
CMD ["npm", "start"]
###Why it's built this way:
Choice
Reason
node:22-alpine
A current, actively supported Node version on a small base image with a reduced attack surface. Prefer a maintained LTS release.
Three stages
Separates dependency install, build, and runtime so the final image carries only what it needs.
Copying package*.json first
Docker caches layers. If your dependencies haven't changed, rebuilds skip the slow install step.
npm ci
Installs exact versions from the lockfile, giving reproducible builds (faster and stricter than npm install).
Non-root user
If the app is ever compromised, the attacker doesn't get root inside the container.
EXPOSE 3000
Documents which port the app listens on. Use 3001 if your app runs there. (just as we did in this case)
Adapt it to your project.
The template above assumes your build outputs to dist/ and npm start runs the production server. Check your package.json scripts and adjust:
Next.js: the build output is .next/, so copy .next, public, and package.json instead of dist. Next.js's output: "standalone" option can shrink the image further.
Vite/React single-page apps: these produce static files in dist/. A common pattern is a final stage using nginx:alpine that serves the static files, in which case you expose port 80 instead.
Express/NestJS APIs: copy your compiled output (often dist/) and set CMD ["node", "dist/main.js"] or similar.
Also make sure your app listens on 0.0.0.0 (not just localhost) and respects the PORT environment variable. Otherwise it won't be reachable from outside the container.
##Step 5: Build the Docker image
From the project root, run:
docker build -t <image-name>:<tag> .
For example I used:
docker build -t wonderkids:latest .
!docker build
-t assigns a name and tag to the image.
The trailing . tells Docker to use the current directory as the build context.
Docker works through each stage and prints progress. When it finishes, confirm the image exists:
docker images
You'll see your image listed with its size. Compare it to a single-stage build and the multi-stage savings will be clear.
Tip: Use meaningful tags such as wonderkids:1.0.0 rather than relying on latest alone, so you can roll back to a known version.
##Step 6: Run the container
Start a container from your image, mapping a port on your machine to the port inside the container:
docker run -p <host-port>:<container-port> --name <container-name> <image-name>:<tag>
For example:
docker run -p 3001:3001 --name wonderkids-app wonderkids:latest
!Docker run
-p 3001:3001 maps host port 3001 to container port 3001. The format is always host:container. To access the app on a different host port, change only the left side: -p 8080:3000 makes it available at localhost.
--name gives the container a friendly name instead of a random one.
Open localhost in your browser to confirm the containerized app is running. Press Ctrl + C to stop it if it's running in the foreground.
Useful variations:
Run in the background (detached mode)
docker run -d -p 3000:3000 --name wonderkids-app wonderkids:latest
Automatically remove the container when it stops
docker run --rm -p 3000:3000 wonderkids:latest
The container has been created on Dockerhub.
###Everyday Docker Commands Useful for Every DevOps Engineer
Task
Command
List running containers
docker ps
List all containers
docker ps -a
View logs
docker logs wonderkids-app
Follow logs live
docker logs -f wonderkids-app
Stop a container
docker stop wonderkids-app
Start a stopped container
docker start wonderkids-app
Remove a container
docker rm wonderkids-app
Remove an image
docker rmi wonderkids:latest
Open a shell inside a container
docker exec -it wonderkids-app sh
###Troubleshooting
"Cannot connect to the Docker daemon." Docker Desktop isn't running. Start it and wait for it to finish initializing.
"Port is already allocated." Something else is using that host port, possibly your local dev server from Part 1 or an older container. Stop it, or map to a different host port (-p 3001:3000).
"Container name already in use." A previous container with that name exists. Remove it with docker rm <container-name> or choose a new name.
Container starts then exits immediately. Check docker logs <container-name>. Common causes are a wrong CMD, a missing build output folder, or missing environment variables.
App runs in the container but the browser can't reach it. Confirm the app binds to 0.0.0.0 rather than 127.0.0.1, and that your -p mapping matches the port the app actually listens on.
Build fails at COPY --from=builder /app/dist. Your framework outputs somewhere other than dist/. Check your build script and update the path.
##Conclusion
We've now taken an application through a complete workflow: cloned it, installed dependencies, verified it locally, and then packaged it as an optimized, secure Docker image that runs identically anywhere Docker does. From here, natural next steps are pushing the image to a registry (Docker Hub or Azure Container Registry), using Docker Compose for multi-service setups, and automating image builds in a CI/CD pipeline. We will complete this in the next article.
6 reactions · 3 comments
Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計
TL;DR: 結論だけ知りたい方へ（今すぐエラーを防ぐ最小コード）
セルの再実行で fatal: destination path 'my-tool' already exists and is not an empty directory. が出て困っている場合は、クローン直前に カレントディレクトリを /content に戻して既存フォルダを削除する 以下の数行をセルの先頭に追加してください。
import os, shutil
# 1. カレントディレクトリを必ず /content に戻す（迷子防止）
os.chdir("/content")
# 2. 既にフォルダが存在していれば削除する（再実行時の衝突防止）
TARGET_DIR = "/content/my-tool"
if os.path.abspath(TARGET_DIR).startswith("/content/drive"):
raise ValueError("❌ TARGET_DIR に Google Drive 配下は指定できません（データ保護）。")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
# 3. クローン実行
!git clone https://github.com/my-org/my-tool.git {TARGET_DIR}
> これだけでは不十分なケースとは？
> 個人利用の単発スクリプトであれば上記で足りますが、「モジュール更新がメモリに残って反映されない（sys.modules キャッシュ問題）」「配布相手の環境でタグ不在や通信切断で止まる」「プライベートリポジトリの認証トークンが残る」 といった実践的な課題を解決するには、追加の考慮が必要です。本稿ではこれらを解決する設計とコードを解説します。
:::note info
【Colab実運用基盤シリーズ】
- 第1弾（データ永続化編）: Google ColabでGoogle Drive上のSQLite / DuckDBが遅い・ロック破損する問題と、ローカルディスクを使ったデータステージング設計
- 第2弾（コード配布編・本作）: Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計
- 関連（前史）: Google Colab × SQLite3 を「壊れる前提」で安定運用する方法（自己修復20行＋AIキャッシュ戦略）
:::
---
はじめに
Google Colaboratory（以下、Colab）は、ブラウザだけで Python コードを実行できるため、オープンソースツールや分析パイプライン（第1弾で触れた株価スクリーナーなど）をチームメンバーや一般ユーザーに配布する手段として広く使われています。
その際、最も一般的な配布方法は「ノートブック内に !git clone https://github.com/... を記述し、リポジトリのコードをそのまま実行させる」という構成です。しかし、この単純な git clone をそのまま配ると、セルの再実行時やリリースのタイミングでエラーが発生することがあり、パイプラインが停止します。
本稿では、Colab 上で GitHub リポジトリのコードと依存ライブラリを確実に立ち上げるための環境セットアップ処理をまとめます。
---
第1章：よくある書き方と、見落としていたこと
GitHub で管理されているコードを Colab 上で利用者に実行させる際、多くのチュートリアルでは以下のようなシンプルなセルが提示されます。
# 初期の典型的なセットアップセル
!git clone -b v1.0.0 https://github.com/my-org/my-tool.git
%cd my-tool
!pip install -r requirements.txt
from src.main import run_pipeline
run_pipeline()
この書き方には、次のような都合のよい想定（思い込み）がありました。
- 「セルを上から順に1度だけ実行する」という想定: セルの再実行や、途中でエラーになってやり直す操作を考えていない
- 「指定したタグは必ず存在する」という想定: リモートにそのタグが確実にプッシュされていると思い込んでいる（push 漏れやタイポを考慮していない）
- 「環境は最初から綺麗である」という想定: %cd によるカレントディレクトリの移動やインストール状態が、再実行時にも都合よく保たれると思い込んでいる
---
第2章：本番運用で起きた3つのトラブル
配布を想定した動作検証や、テスト配布を行った際、以下のトラブルに遭遇しました。
1. セル再実行時のディレクトリ衝突エラー
エラー発生やパラメータ変更で利用者がセットアップセルをもう一度実行した際、Git が衝突エラーを起こして停止しました。
fatal: destination path 'my-tool' already exists and is not an empty directory.
利用者は手動でファイルを消す方法がわからず、ランタイムの全初期化を強いられます。
2. リリース直後のタグ不一致によるエラー中断
新バージョン（例: v1.3.1）のリリース告知直後に、利用者がクローンを実行すると以下のエラーで停止しました。
fatal: Remote branch v1.3.1 not found in upstream origin
この原因の多くは、開発者側で git push 時にタグの送信を忘れていた（git push --tags の実行漏れ）か、ノートブック側のパラメータに入力したタグ名のタイポでした。このような人為的ミスや指定の不整合により、パイプライン全体が中断し、手動対応が必要になりました。
3. 作業ディレクトリ消失によるモジュール見失い
セル内で %cd my-tool を実行した状態で、前述のエラー復旧や再実行のために外側から shutil.rmtree("my-tool") が走ると、Python プロセスのカレントディレクトリが存在しない無効なパスを指す状態になります。その結果、以降の相対パス操作やファイル読み込みで FileNotFoundError が発生しました。また、sys.modules に過去の古いモジュールがキャッシュされ、再取得したコードがメモリ上に反映されない不整合も起きました。
---
第3章：なぜ単純な git clone では失敗するのか
起きていた問題の原因を掘り下げると、Colab という対話的実行環境の特性に対する考慮が不足していました。
1. セルの再実行（やり直し）を考慮していない
Colab は利用者が試行錯誤しながらセルを何度も再実行する環境です。クローン先にフォルダが既に存在するかどうかを気にせず、常に同じ名前で git clone を実行するコードは、やり直し（再実行）に耐えられません。
2. タグの直書きによるエラー停止
特定のタグ（--branch v1.3.1）をコード内に直書きしていると、開発者のタグ push 忘れや、利用者のタイポが1箇所あるだけでスクリプト全体が停止してしまいます。通信エラーなのか、タグが存在しないのかを判別できず、代替ブランチへの切り替えもできない構成だったため、すぐに処理が止まってしまっていました。
3. カレントディレクトリやモジュールキャッシュがプロセス内に残り続ける問題
Colab ノートブックのカーネルは、単一の持続的な Python プロセス上で対話的に実行されます。シェルコマンドの %cd や import による状態変更はプロセス全体に残り続けるため、以下のトラブルを引き起こします。
1. 滞在中のフォルダを削除してしまう（カレントディレクトリの喪失）:
セル内で %cd my-tool を実行した後に、再クローン目的で外側から shutil.rmtree("my-tool") を実行すると、Python プロセス自身がいまいる場所（カレントディレクトリ）がディスク上から消えてしまいます。この状態になると、以降の相対パス指定やファイルの読み書きで FileNotFoundError が発生します。
2. sys.modules によるモジュールキャッシュの残存:
Python のインポート機構（importlib）は、一度読み込んだモジュールをプロセス空間の辞書 sys.modules にキャッシュします。ファイルシステム上の .py ファイルを新バージョンでクローンし直しても、Python はディスクを再走査せずメモリ上の古いモジュールを返し続けます。その結果、セルの再実行を行っても最新のバグ修正が反映されず、新設された関数が見つからない AttributeError や古いロジックによる不整合が発生します。
---
第4章：どう設計し直したか（再実行に強い環境セットアップ）
これらの問題を踏まえ、利用者がセルの再実行を行っても安定して起動するセットアップ処理を設計し直しました。
設計の基本方針
- 作業ディレクトリのルート固定（/content に戻す）: 操作の直前に必ずカレントディレクトリを /content に戻す。既存の展開先ディレクトリは事前にクリーンアップし、常にクリーンな状態からクローンを開始する
- 最新コミットだけの取得（git clone --depth 1）: 過去のコミット履歴をダウンロードせず、指定した最新のスナップショットのみを取得して通信量と所要時間を抑える
- タグの事前確認と main ブランチへの切り替え（代替取得）: git ls-remote でタグの存在を確認し、見つからない場合はフラグ（許可時のみ）に応じて main ブランチに切り替えて取得を継続する
- 動的インポートとモジュールキャッシュのリセット: sys.modules 内の自作パッケージを明示的にクリアし、再実行時でも新しくクローンされたコードをメモリに再ロードする
flowchart TD
Start["Step 1 実行開始"] --> CD["os.chdir('/content')<br/>(カレントを /content に戻す)"]
CD --> CheckDir{"TARGET_DIR が<br/>既に存在するか？"}
CheckDir -- "Yes" --> Clean["shutil.rmtree()<br/>(既存ディレクトリを事前消去)"]
CheckDir -- "No" --> Verify
Clean --> Verify["check_tag() でリモート確認"]
Verify -- "通信エラー (error)" --> Retry["指数バックオフで再試行 (最大3回)"]
Retry --> Verify
Verify -- "再試行上限超過" --> ErrorExit["RuntimeError で明示的中断<br/>(GitHub接続障害案内)"]
Verify -- "タグ存在 (found)" --> CloneTag["Attempt: git clone --depth 1<br/>--branch TARGET_TAG (指定タグ)"]
Verify -- "タグ不在 (not_found)" --> CheckAllow{"ALLOW_MAIN_FALLBACK<br/>が有効か？"}
CheckAllow -- "No (無効)" --> TagExit["RuntimeError で中断<br/>(タグ名確認案内)"]
CheckAllow -- "Yes (有効)" --> CloneMain["警告出力 ＆ ブランチを 'main' に切替<br/>git clone --branch main"]
CloneTag --> Pip["sys.executable -m pip install -q -r requirements.txt"]
CloneMain --> Pip
Pip --> Reload["sys.modules から自作パッケージを破棄<br/>(最新モジュールの強制再読込)"]
Reload --> Ready["環境セットアップ完了 (次Stepへ)"]
style Start fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
style Pip fill:#f0fff0,stroke:#2a2,stroke-width:1px,color:#333
style Ready fill:#d0ffd0,stroke:#2a2,stroke-width:2px,color:#333
style ErrorExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333
style TagExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333
---
第5章：実装のポイントとコード
ここまでに整理した設計を、Google Colab 上でそのまま動作確認できるようにまとめた動作確認用のセットアップコードです。
ノートブックの1つのセルに貼り付け、上部のパラメータ（REPO_URL やタグ名）をご自身のリポジトリに合わせて設定して実行してください。クローン、依存関係のインストール、モジュールキャッシュの破棄、パス解決までが一気通貫で完了します。
# @title 【Step 1】環境セットアップ
# @markdown 配布したいリポジトリの情報を設定してください
REPO_URL = "https://github.com/my-org/my-tool.git" # @param {type:"string"}
TARGET_TAG = "v1.3.1" # @param {type:"string"}
ALLOW_MAIN_FALLBACK = False # @param {type:"boolean"}
# ↑ タグ不在時に最新開発版 (main) での実行を許容する場合は True にチェック
TARGET_DIR = "/content/my-tool"
PACKAGE_NAME = "src" # 読み込むパッケージフォルダ名（例: src または my_tool）
import importlib
import os
import re
import shutil
import subprocess
import sys
import time
# 1. 作業ディレクトリを必ず /content に戻す
os.chdir("/content")
# 2. 既存ディレクトリが存在する場合は削除 (再実行時の衝突防止)
# ※Drive配下の誤削除を防ぐためガードを配置
if os.path.abspath(TARGET_DIR).startswith("/content/drive"):
raise ValueError("❌ TARGET_DIR に Google Drive 配下のパスは指定できません（データ保護のため）。")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
# 3. リモートタグの事前確認（3状態判定と指数バックオフ）
def check_tag(repo_url: str, tag: str, retries: int = 3) -> str:
"""'found' / 'not_found' / 'error' を返す。通信エラーは指数バックオフで再試行"""
git_env = os.environ.copy()
git_env["GIT_TERMINAL_PROMPT"] = "0" # 認証プロンプトによる無応答停止を防止
for i in range(retries):
try:
res = subprocess.run(
["git", "ls-remote", "--exit-code", repo_url, f"refs/tags/{tag}"],
capture_output=True,
text=True,
timeout=10,
env=git_env,
)
if res.returncode == 0:
return "found"
if res.returncode == 2: # 通信成功・該当refなし
return "not_found"
except (subprocess.TimeoutExpired, subprocess.CalledProcessError):
pass
time.sleep(2 ** i)
return "error"
status = check_tag(REPO_URL, TARGET_TAG)
if status == "error":
raise RuntimeError(
"❌ リポジトリが見つからないか、GitHub に接続できません。\n"
"上部の 'REPO_URL' を実際のリポジトリURLに書き換えて再実行してください。"
)
if status == "not_found":
if not ALLOW_MAIN_FALLBACK:
raise RuntimeError(f"❌ タグ '{TARGET_TAG}' が見つかりません。タグ名を確認してください。")
print(f"⚠️ タグ '{TARGET_TAG}' がないため main で続行します（未検証コードの可能性あり）。")
target_to_clone = "main"
else:
target_to_clone = TARGET_TAG
# 4. 最新コミットだけの取得 (--depth 1)
clone_success = False
git_env = os.environ.copy()
git_env["GIT_TERMINAL_PROMPT"] = "0"
for attempt in range(1, 3):
try:
print(f"📥 リポジトリを取得中 (attempt {attempt}/2, target: {target_to_clone})...")
subprocess.run(
["git", "clone", "--depth", "1", "--branch", target_to_clone, REPO_URL, TARGET_DIR],
check=True,
capture_output=True,
text=True,
env=git_env,
)
clone_success = True
break
except subprocess.CalledProcessError as e:
# トークン等を含むコマンド全文ではなく、標準エラーのみを安全に出力（URLトークンは伏字化）
raw_msg = e.stderr.strip() if e.stderr else "クローン処理に失敗しました"
err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg)
print(f"⚠️ Git クローン警告 (attempt {attempt}): {err_msg}")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
time.sleep(2)
if not clone_success:
raise RuntimeError("❌ リポジトリ取得に失敗しました。ネットワーク接続または GitHub の稼働状況を確認してください。")
# 5. 依存ライブラリの導入 (カレントディレクトリを移動せず、sys.executable と絶対パスで指定)
req_path = os.path.join(TARGET_DIR, "requirements.txt")
if os.path.exists(req_path):
try:
print("📦 依存ライブラリをインストール中...")
subprocess.run(
[sys.executable, "-m", "pip", "install", "-q", "-r", req_path],
check=True,
capture_output=True,
text=True,
)
except subprocess.CalledProcessError as e:
err_msg = e.stderr.strip() if e.stderr else "pip インストールに失敗しました"
raise RuntimeError(f"❌ 依存ライブラリのインストールに失敗しました: {err_msg}") from None
# 6. 自作モジュールのキャッシュを破棄 (再実行時の古いコード混入を防止)
for mod in list(sys.modules.keys()):
if mod == PACKAGE_NAME or mod.startswith(f"{PACKAGE_NAME}."):
del sys.modules[mod]
importlib.invalidate_caches() # ファインダーの内部ファイル検索キャッシュをリセット
# パスを追加してインポートを有効化
if TARGET_DIR not in sys.path:
sys.path.insert(0, TARGET_DIR)
# 7. エントリポイントのインポート確認
try:
main_mod = importlib.import_module(f"{PACKAGE_NAME}.main")
run_pipeline = getattr(main_mod, "run_pipeline", None)
print(f"✅ エントリポイント（{PACKAGE_NAME}.main）のインポートに成功しました。")
except ModuleNotFoundError as e:
# パッケージまたは main モジュール自体が存在しない場合のみスキップ
if e.name in (PACKAGE_NAME, f"{PACKAGE_NAME}.main"):
print(f"ℹ️ {PACKAGE_NAME}.main が見つからないため、インポートをスキップしました（ご自身のリポジトリ構造に合わせて書き換えてください）。")
else:
# 依存ライブラリ等の読み込み失敗は本物のエラーとして再送出
raise
print("✅ セットアップが完了しました。")
💡 このコードを配布用ノートブックに組み込む手順
1. 新しいノートブックを作成: 最初のセルに上記コードをそのまま貼り付けます。
2. パラメータを3箇所変更:
- REPO_URL: ご自身の GitHub リポジトリ URL（例: https://github.com/your-org/your-repo.git）
- TARGET_TAG: 配布したい安定版のタグ名（例: v1.0.0）
- PACKAGE_NAME: リポジトリ内の自作モジュール名（例: src や my_tool）
3. フォームを整えてコードを隠す（任意）:
- セルの右上メニュー（縦三点リーダー）から「フォーム」➔「コードを非表示」を選択すると、一般利用者の画面には生コードが隠れ、タイトルと入力フォーム（実行ボタン）だけが表示された綺麗な UI として配布できます。
4. 後続セル（Step 2）から実行:
- 次のセル（Step 2）を作成し、from {PACKAGE_NAME}.main import run_pipeline; run_pipeline()（例: from src.main import run_pipeline; run_pipeline()）のように呼び出せば、クローンされたコードがそのまま動作します。
:::note info
パッケージ名（ここでは src）は、サードパーティ製ライブラリと重複しない固有のパッケージ名（例: my_tool）にしておくと、名前空間の衝突をより確実に防ぐことができます。
:::
2. コラム：プライベートリポジトリを安全に扱うには？
もしコードを一般公開せず、チームメンバーや特定利用者向けに配布する場合、トークンをハードコードしてはいけません。Colab 標準の 「Secrets（🔑）」機能 を使って環境変数経由でトークンを注入し、クローン完了後にローカルの Git 設定からトークンを抹消します。
# プライベートリポジトリの場合の安全なトークン注入と痕跡消去
import re
import subprocess
from google.colab import userdata
try:
gh_token = userdata.get("GITHUB_TOKEN")
auth_repo_url = f"https://x-access-token:{gh_token}@github.com/my-org/private-tool.git"
except Exception:
raise RuntimeError("🔑 左メニューの鍵アイコン (Secrets) から 'GITHUB_TOKEN' を設定してください。")
# クローン実行（エラー出力時にトークンが露出しないよう URL 部分を伏字化）
try:
subprocess.run(
["git", "clone", "--depth", "1", "--branch", TARGET_TAG, auth_repo_url, TARGET_DIR],
check=True,
capture_output=True,
text=True,
)
# クローン成功後、ローカル .git/config 内の URL からトークンを即時除去
clean_repo_url = "https://github.com/my-org/private-tool.git"
subprocess.run(
["git", "-C", TARGET_DIR, "remote", "set-url", "origin", clean_repo_url],
check=True,
)
except subprocess.CalledProcessError as e:
raw_msg = e.stderr.strip() if e.stderr else "認証またはリポジトリの取得に失敗しました"
err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg)
raise RuntimeError(f"❌ プライベートリポジトリの取得エラー: {err_msg}") from None
---
第6章：動かしてみた結果とベンチマーク
このセットアップ処理の導入により、運用の安定性が向上しました。
1. 起動時間とエラー耐性の比較
数百コミット規模のリポジトリを Colab 無料枠でクローンした筆者環境での実測値です。
| 項目 / 状態 | 通常の !git clone | 改善したセットアップ | 改善効果 |
| :--------------------------- | :---------------------------: | :----------------------------------: | :------------------------- |
| 初動の転送時間 | 約 5〜10 秒（全履歴取得時） | 約 2〜4 秒（--depth 1） | 初動の短縮 |
| 同一セルの再実行 | already exists でエラー停止 | 自動消去・再取得で衝突なし | 冪等性の担保 |
| タグ push 漏れ・タイポ時 | エラーによる処理中断 | 検証を経て許可時のみ main 退避 | タグ不在時に main 代替取得 |
| 自作モジュール更新 | 再起動しないと旧コード参照 | sys.modules 破棄で即反映 | コード不整合の防止 |
2. 同一ランタイムでのセル再実行に対する冪等性と復旧
セッションが完全に切断された場合は Colab の VM ごと初期化されるため前回の残骸は残りませんが、実際の運用で頻繁に発生するのは「エラー発生後やパラメータ変更時に、同一ランタイム上でセルをもう一度実行する」ケースです。
セットアップコードの冒頭でカレントディレクトリを /content に戻し、既存フォルダを消去することで、同じセッション内でのやり直し時にディレクトリの衝突や、削除済みフォルダ内に留まって迷子になる問題を防ぎます。
※なお、pip install の実行途中で処理を強制中断した場合、site-packages 配下に不完全なパッケージが残存する可能性があります。その場合はノートブックのメニューからランタイムの再起動（Restart session）を行うか、pip install --force-reinstall を実行して修復します。
3. 今回の設計が持つトレードオフと運用上の制約
本設計は「一般利用者の手元で止まりにくく、安全に再実行できること」を優先しています。そのため、以下のトレードオフ（何を得て、何を妥協したか）が存在します。
- 【起動速度 vs Git履歴の閲覧性（--depth 1 の代償）】最新コミットに絞ることで履歴追跡は切り捨てる:
--depth 1 によって数秒での高速クローンを実現した代償として、過去のコミット履歴（git log や git diff、過去リビジョンへのチェックアウト）は利用できません。コードを配布して実行させる用途に特化し、開発者向けの履歴閲覧性は意図して捨てています
- 【可用性（止まらないこと） vs コードの再現性】main への代替切り替えは未検証コードのリスクを伴う:
タグ不在時に main ブランチへ自動退避して処理を継続させる選択肢を用意した代償として、開発途中の未検証コードが動いてしまうリスクを負います。そのため本設計では ALLOW_MAIN_FALLBACK をデフォルトで False とし、利用者がリスクを承知の上で明示的に有効化した場合のみ main へ切り替える安全弁を設けています
- 【高速な再読込 vs C拡張モジュールの更新限界】インメモリ破棄で扱えるのは純粋な Python コードに限定される:
ランタイム再起動を挟まずに sys.modules の破棄で最新コードを即時反映させる軽快さを取った代償として、numpy や torch など一度メモリに共有ライブラリ（.so）がロードされたバイナリ拡張パッケージの更新は反映できません。これらをアップグレードした場合は、OS プロセス自体の再起動（ランタイム再起動）が必要です
- 【安全なトークン注入 vs ログ出力・プロセス一覧からの漏洩リスク】例外メッセージのサニタイズが必要になる:
Secrets を用いてプライベートリポジトリを安全にクローンできる利便性を取った代償として、例外発生時にトークンが露出しないよう stderr のサニタイズ（伏字化）が必要です。また、クローン実行中は OS のプロセス一覧（ps）から一時的に引数のトークンが見える点にも留意が必要です
---
結び
Google Colab 上でリポジトリのコードを動かす作業は、ローカル開発環境での git clone とは前提が異なります。
「作業ディレクトリを /content に戻す」「既存フォルダの事前クリーンアップ」「--depth 1 による最新取得」「タグ不在時の main 代替取得」「モジュールキャッシュの破棄」という一連の対策をセットアップセルに組み込むことで、ボタン1つで安定して起動できる配布基盤を構築できます。
本稿で確立したコード展開フローと、第1弾（データ永続化編）で設計した 2 層ストレージフローを組み合わせることで、ノートブック全体のライフサイクルは以下のように整理されます。
> 【Step 1】環境セットアップ（本稿：コード展開・依存解決）
> ↓
> 【Step 2】Pull Phase（第1弾：永続層 Drive から作業層ローカルディスクへデータステージング展開）
> ↓
> 【Step 3】処理実行（ローカルディスク上での高速バッチ・クエリ処理）
> ↓
> 【Step 4】Push Phase（第1弾：成果物先行・DB末尾置換でデータステージング書き戻し ＆ アンマウント）
2 つの設計を組み合わせることで、対話的なノートブックであっても運用に耐えうる安定したバッチ実行基盤として活用できるようになります。
この記事が参考になったら、ストックしておくとシリーズの続きを追いやすくなります。
質問や、別の環境で試した結果はコメントで教えてください。
---
> 免責事項（Disclaimer）
> 本記事に掲載されているコードや設定例は筆者の検証環境に基づくものであり、外部サービス（Google Colab、GitHub 等）の仕様変更や実行環境の違い等によって生じたいかなる損害についても責任を負いかねます。実際の運用や設定はご自身の責任において確認の上で行ってください。
Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計
TL;DR: 結論だけ知りたい方へ（今すぐエラーを防ぐ最小コード）
セルの再実行で fatal: destination path 'my-tool' already exists and is not an empty directory. が出て困っている場合は、クローン直前に カレントディレクトリを /content に戻して既存フォルダを削除する 以下の数行をセルの先頭に追加してください。
import os, shutil
# 1. カレントディレクトリを必ず /content に戻す（迷子防止）
os.chdir("/content")
# 2. 既にフォルダが存在していれば削除する（再実行時の衝突防止）
TARGET_DIR = "/content/my-tool"
if os.path.abspath(TARGET_DIR).startswith("/content/drive"):
raise ValueError("❌ TARGET_DIR に Google Drive 配下は指定できません（データ保護）。")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
# 3. クローン実行
!git clone https://github.com/my-org/my-tool.git {TARGET_DIR}
> これだけでは不十分なケースとは？
> 個人利用の単発スクリプトであれば上記で足りますが、「モジュール更新がメモリに残って反映されない（sys.modules キャッシュ問題）」「配布相手の環境でタグ不在や通信切断で止まる」「プライベートリポジトリの認証トークンが残る」 といった実践的な課題を解決するには、追加の考慮が必要です。本稿ではこれらを解決する設計とコードを解説します。
:::note info
【Colab実運用基盤シリーズ】
- 第1弾（データ永続化編）: Google ColabでGoogle Drive上のSQLite / DuckDBが遅い・ロック破損する問題と、ローカルディスクを使ったデータステージング設計
- 第2弾（コード配布編・本作）: Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計
:::
---
はじめに
Google Colaboratory（以下、Colab）は、ブラウザだけで Python コードを実行できるため、オープンソースツールや分析パイプライン（第1弾で触れた株価スクリーナーなど）をチームメンバーや一般ユーザーに配布する手段として広く使われています。
その際、最も一般的な配布方法は「ノートブック内に !git clone https://github.com/... を記述し、リポジトリのコードをそのまま実行させる」という構成です。しかし、この単純な git clone をそのまま配ると、セルの再実行時やリリースのタイミングでエラーが発生することがあり、パイプラインが停止します。
本稿では、Colab 上で GitHub リポジトリのコードと依存ライブラリを確実に立ち上げるための環境セットアップ処理をまとめます。
---
第1章：よくある書き方と、見落としていたこと
GitHub で管理されているコードを Colab 上で利用者に実行させる際、多くのチュートリアルでは以下のようなシンプルなセルが提示されます。
# 初期の典型的なセットアップセル
!git clone -b v1.0.0 https://github.com/my-org/my-tool.git
%cd my-tool
!pip install -r requirements.txt
from src.main import run_pipeline
run_pipeline()
この書き方には、次のような都合のよい想定（思い込み）がありました。
- 「セルを上から順に1度だけ実行する」という想定: セルの再実行や、途中でエラーになってやり直す操作を考えていない
- 「指定したタグは必ず存在する」という想定: リモートにそのタグが確実にプッシュされていると思い込んでいる（push 漏れやタイポを考慮していない）
- 「環境は最初から綺麗である」という想定: %cd によるカレントディレクトリの移動やインストール状態が、再実行時にも都合よく保たれると思い込んでいる
---
第2章：本番運用で起きた3つのトラブル
配布を想定した動作検証や、テスト配布を行った際、以下のトラブルに遭遇しました。
1. セル再実行時のディレクトリ衝突エラー
エラー発生やパラメータ変更で利用者がセットアップセルをもう一度実行した際、Git が衝突エラーを起こして停止しました。
fatal: destination path 'my-tool' already exists and is not an empty directory.
利用者は手動でファイルを消す方法がわからず、ランタイムの全初期化を強いられます。
2. リリース直後のタグ不一致によるエラー中断
新バージョン（例: v1.3.1）のリリース告知直後に、利用者がクローンを実行すると以下のエラーで停止しました。
fatal: Remote branch v1.3.1 not found in upstream origin
この原因の多くは、開発者側で git push 時にタグの送信を忘れていた（git push --tags の実行漏れ）か、ノートブック側のパラメータに入力したタグ名のタイポでした。このような人為的ミスや指定の不整合により、パイプライン全体が中断し、手動対応が必要になりました。
3. 作業ディレクトリ消失によるモジュール見失い
セル内で %cd my-tool を実行した状態で、前述のエラー復旧や再実行のために外側から shutil.rmtree("my-tool") が走ると、Python プロセスのカレントディレクトリが存在しない無効なパスを指す状態になります。その結果、以降の相対パス操作やファイル読み込みで FileNotFoundError が発生しました。また、sys.modules に過去の古いモジュールがキャッシュされ、再取得したコードがメモリ上に反映されない不整合も起きました。
---
第3章：なぜ単純な git clone では失敗するのか
起きていた問題の原因を掘り下げると、Colab という対話的実行環境の特性に対する考慮が不足していました。
1. セルの再実行（やり直し）を考慮していない
Colab は利用者が試行錯誤しながらセルを何度も再実行する環境です。クローン先にフォルダが既に存在するかどうかを気にせず、常に同じ名前で git clone を実行するコードは、やり直し（再実行）に耐えられません。
2. タグの直書きによるエラー停止
特定のタグ（--branch v1.3.1）をコード内に直書きしていると、開発者のタグ push 忘れや、利用者のタイポが1箇所あるだけでスクリプト全体が停止してしまいます。通信エラーなのか、タグが存在しないのかを判別できず、代替ブランチへの切り替えもできない構成だったため、すぐに処理が止まってしまっていました。
3. カレントディレクトリやモジュールキャッシュがプロセス内に残り続ける問題
Colab ノートブックのカーネルは、単一の持続的な Python プロセス上で対話的に実行されます。シェルコマンドの %cd や import による状態変更はプロセス全体に残り続けるため、以下のトラブルを引き起こします。
1. 滞在中のフォルダを削除してしまう（カレントディレクトリの喪失）:
セル内で %cd my-tool を実行した後に、再クローン目的で外側から shutil.rmtree("my-tool") を実行すると、Python プロセス自身がいまいる場所（カレントディレクトリ）がディスク上から消えてしまいます。この状態になると、以降の相対パス指定やファイルの読み書きで FileNotFoundError が発生します。
2. sys.modules によるモジュールキャッシュの残存:
Python のインポート機構（importlib）は、一度読み込んだモジュールをプロセス空間の辞書 sys.modules にキャッシュします。ファイルシステム上の .py ファイルを新バージョンでクローンし直しても、Python はディスクを再走査せずメモリ上の古いモジュールを返し続けます。その結果、セルの再実行を行っても最新のバグ修正が反映されず、新設された関数が見つからない AttributeError や古いロジックによる不整合が発生します。
---
第4章：どう設計し直したか（再実行に強い環境セットアップ）
これらの問題を踏まえ、利用者がセルの再実行を行っても安定して起動するセットアップ処理を設計し直しました。
設計の基本方針
- 作業ディレクトリのルート固定（/content に戻す）: 操作の直前に必ずカレントディレクトリを /content に戻す。既存の展開先ディレクトリは事前にクリーンアップし、常にクリーンな状態からクローンを開始する
- 最新コミットだけの取得（git clone --depth 1）: 過去のコミット履歴をダウンロードせず、指定した最新のスナップショットのみを取得して通信量と所要時間を抑える
- タグの事前確認と main ブランチへの切り替え（代替取得）: git ls-remote でタグの存在を確認し、見つからない場合はフラグ（許可時のみ）に応じて main ブランチに切り替えて取得を継続する
- 動的インポートとモジュールキャッシュのリセット: sys.modules 内の自作パッケージを明示的にクリアし、再実行時でも新しくクローンされたコードをメモリに再ロードする
flowchart TD
Start["Step 1 実行開始"] --> CD["os.chdir('/content')<br/>(カレントを /content に戻す)"]
CD --> CheckDir{"TARGET_DIR が<br/>既に存在するか？"}
CheckDir -- "Yes" --> Clean["shutil.rmtree()<br/>(既存ディレクトリを事前消去)"]
CheckDir -- "No" --> Verify
Clean --> Verify["check_tag() でリモート確認"]
Verify -- "通信エラー (error)" --> Retry["指数バックオフで再試行 (最大3回)"]
Retry --> Verify
Verify -- "再試行上限超過" --> ErrorExit["RuntimeError で明示的中断<br/>(GitHub接続障害案内)"]
Verify -- "タグ存在 (found)" --> CloneTag["Attempt: git clone --depth 1<br/>--branch TARGET_TAG (指定タグ)"]
Verify -- "タグ不在 (not_found)" --> CheckAllow{"ALLOW_MAIN_FALLBACK<br/>が有効か？"}
CheckAllow -- "No (無効)" --> TagExit["RuntimeError で中断<br/>(タグ名確認案内)"]
CheckAllow -- "Yes (有効)" --> CloneMain["警告出力 ＆ ブランチを 'main' に切替<br/>git clone --branch main"]
CloneTag --> Pip["sys.executable -m pip install -q -r requirements.txt"]
CloneMain --> Pip
Pip --> Reload["sys.modules から自作パッケージを破棄<br/>(最新モジュールの強制再読込)"]
Reload --> Ready["環境セットアップ完了 (次Stepへ)"]
style Start fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333
style Pip fill:#f0fff0,stroke:#2a2,stroke-width:1px,color:#333
style Ready fill:#d0ffd0,stroke:#2a2,stroke-width:2px,color:#333
style ErrorExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333
style TagExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333
---
第5章：実装のポイントとコード
ここまでに整理した設計を、Google Colab 上でそのまま動作確認できるようにまとめた動作確認用のセットアップコードです。
ノートブックの1つのセルに貼り付け、上部のパラメータ（REPO_URL やタグ名）をご自身のリポジトリに合わせて設定して実行してください。クローン、依存関係のインストール、モジュールキャッシュの破棄、パス解決までが一気通貫で完了します。
# @title 【Step 1】環境セットアップ
# @markdown 配布したいリポジトリの情報を設定してください
REPO_URL = "https://github.com/my-org/my-tool.git" # @param {type:"string"}
TARGET_TAG = "v1.3.1" # @param {type:"string"}
ALLOW_MAIN_FALLBACK = False # @param {type:"boolean"}
# ↑ タグ不在時に最新開発版 (main) での実行を許容する場合は True にチェック
TARGET_DIR = "/content/my-tool"
PACKAGE_NAME = "src" # 読み込むパッケージフォルダ名（例: src または my_tool）
import importlib
import os
import re
import shutil
import subprocess
import sys
import time
# 1. 作業ディレクトリを必ず /content に戻す
os.chdir("/content")
# 2. 既存ディレクトリが存在する場合は削除 (再実行時の衝突防止)
# ※Drive配下の誤削除を防ぐためガードを配置
if os.path.abspath(TARGET_DIR).startswith("/content/drive"):
raise ValueError("❌ TARGET_DIR に Google Drive 配下のパスは指定できません（データ保護のため）。")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
# 3. リモートタグの事前確認（3状態判定と指数バックオフ）
def check_tag(repo_url: str, tag: str, retries: int = 3) -> str:
"""'found' / 'not_found' / 'error' を返す。通信エラーは指数バックオフで再試行"""
git_env = os.environ.copy()
git_env["GIT_TERMINAL_PROMPT"] = "0" # 認証プロンプトによる無応答停止を防止
for i in range(retries):
try:
res = subprocess.run(
["git", "ls-remote", "--exit-code", repo_url, f"refs/tags/{tag}"],
capture_output=True,
text=True,
timeout=10,
env=git_env,
)
if res.returncode == 0:
return "found"
if res.returncode == 2: # 通信成功・該当refなし
return "not_found"
except (subprocess.TimeoutExpired, subprocess.CalledProcessError):
pass
time.sleep(2 ** i)
return "error"
status = check_tag(REPO_URL, TARGET_TAG)
if status == "error":
raise RuntimeError(
"❌ リポジトリが見つからないか、GitHub に接続できません。\n"
"上部の 'REPO_URL' を実際のリポジトリURLに書き換えて再実行してください。"
)
if status == "not_found":
if not ALLOW_MAIN_FALLBACK:
raise RuntimeError(f"❌ タグ '{TARGET_TAG}' が見つかりません。タグ名を確認してください。")
print(f"⚠️ タグ '{TARGET_TAG}' がないため main で続行します（未検証コードの可能性あり）。")
target_to_clone = "main"
else:
target_to_clone = TARGET_TAG
# 4. 最新コミットだけの取得 (--depth 1)
clone_success = False
git_env = os.environ.copy()
git_env["GIT_TERMINAL_PROMPT"] = "0"
for attempt in range(1, 3):
try:
print(f"📥 リポジトリを取得中 (attempt {attempt}/2, target: {target_to_clone})...")
subprocess.run(
["git", "clone", "--depth", "1", "--branch", target_to_clone, REPO_URL, TARGET_DIR],
check=True,
capture_output=True,
text=True,
env=git_env,
)
clone_success = True
break
except subprocess.CalledProcessError as e:
# トークン等を含むコマンド全文ではなく、標準エラーのみを安全に出力（URLトークンは伏字化）
raw_msg = e.stderr.strip() if e.stderr else "クローン処理に失敗しました"
err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg)
print(f"⚠️ Git クローン警告 (attempt {attempt}): {err_msg}")
if os.path.exists(TARGET_DIR):
shutil.rmtree(TARGET_DIR, ignore_errors=True)
time.sleep(2)
if not clone_success:
raise RuntimeError("❌ リポジトリ取得に失敗しました。ネットワーク接続または GitHub の稼働状況を確認してください。")
# 5. 依存ライブラリの導入 (カレントディレクトリを移動せず、sys.executable と絶対パスで指定)
req_path = os.path.join(TARGET_DIR, "requirements.txt")
if os.path.exists(req_path):
try:
print("📦 依存ライブラリをインストール中...")
subprocess.run(
[sys.executable, "-m", "pip", "install", "-q", "-r", req_path],
check=True,
capture_output=True,
text=True,
)
except subprocess.CalledProcessError as e:
err_msg = e.stderr.strip() if e.stderr else "pip インストールに失敗しました"
raise RuntimeError(f"❌ 依存ライブラリのインストールに失敗しました: {err_msg}") from None
# 6. 自作モジュールのキャッシュを破棄 (再実行時の古いコード混入を防止)
for mod in list(sys.modules.keys()):
if mod == PACKAGE_NAME or mod.startswith(f"{PACKAGE_NAME}."):
del sys.modules[mod]
importlib.invalidate_caches() # ファインダーの内部ファイル検索キャッシュをリセット
# パスを追加してインポートを有効化
if TARGET_DIR not in sys.path:
sys.path.insert(0, TARGET_DIR)
# 7. エントリポイントのインポート確認
try:
main_mod = importlib.import_module(f"{PACKAGE_NAME}.main")
run_pipeline = getattr(main_mod, "run_pipeline", None)
print(f"✅ エントリポイント（{PACKAGE_NAME}.main）のインポートに成功しました。")
except ModuleNotFoundError as e:
# パッケージまたは main モジュール自体が存在しない場合のみスキップ
if e.name in (PACKAGE_NAME, f"{PACKAGE_NAME}.main"):
print(f"ℹ️ {PACKAGE_NAME}.main が見つからないため、インポートをスキップしました（ご自身のリポジトリ構造に合わせて書き換えてください）。")
else:
# 依存ライブラリ等の読み込み失敗は本物のエラーとして再送出
raise
print("✅ セットアップが完了しました。")
💡 このコードを配布用ノートブックに組み込む手順
1. 新しいノートブックを作成: 最初のセルに上記コードをそのまま貼り付けます。
2. パラメータを3箇所変更:
- REPO_URL: ご自身の GitHub リポジトリ URL（例: https://github.com/your-org/your-repo.git）
- TARGET_TAG: 配布したい安定版のタグ名（例: v1.0.0）
- PACKAGE_NAME: リポジトリ内の自作モジュール名（例: src や my_tool）
3. フォームを整えてコードを隠す（任意）:
- セルの右上メニュー（縦三点リーダー）から「フォーム」➔「コードを非表示」を選択すると、一般利用者の画面には生コードが隠れ、タイトルと入力フォーム（実行ボタン）だけが表示された綺麗な UI として配布できます。
4. 後続セル（Step 2）から実行:
- 次のセル（Step 2）を作成し、from {PACKAGE_NAME}.main import run_pipeline; run_pipeline()（例: from src.main import run_pipeline; run_pipeline()）のように呼び出せば、クローンされたコードがそのまま動作します。
:::note info
パッケージ名（ここでは src）は、サードパーティ製ライブラリと重複しない固有のパッケージ名（例: my_tool）にしておくと、名前空間の衝突をより確実に防ぐことができます。
:::
2. コラム：プライベートリポジトリを安全に扱うには？
もしコードを一般公開せず、チームメンバーや特定利用者向けに配布する場合、トークンをハードコードしてはいけません。Colab 標準の 「Secrets（🔑）」機能 を使って環境変数経由でトークンを注入し、クローン完了後にローカルの Git 設定からトークンを抹消します。
# プライベートリポジトリの場合の安全なトークン注入と痕跡消去
import re
import subprocess
from google.colab import userdata
try:
gh_token = userdata.get("GITHUB_TOKEN")
auth_repo_url = f"https://x-access-token:{gh_token}@github.com/my-org/private-tool.git"
except Exception:
raise RuntimeError("🔑 左メニューの鍵アイコン (Secrets) から 'GITHUB_TOKEN' を設定してください。")
# クローン実行（エラー出力時にトークンが露出しないよう URL 部分を伏字化）
try:
subprocess.run(
["git", "clone", "--depth", "1", "--branch", TARGET_TAG, auth_repo_url, TARGET_DIR],
check=True,
capture_output=True,
text=True,
)
# クローン成功後、ローカル .git/config 内の URL からトークンを即時除去
clean_repo_url = "https://github.com/my-org/private-tool.git"
subprocess.run(
["git", "-C", TARGET_DIR, "remote", "set-url", "origin", clean_repo_url],
check=True,
)
except subprocess.CalledProcessError as e:
raw_msg = e.stderr.strip() if e.stderr else "認証またはリポジトリの取得に失敗しました"
err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg)
raise RuntimeError(f"❌ プライベートリポジトリの取得エラー: {err_msg}") from None
---
第6章：動かしてみた結果とベンチマーク
このセットアップ処理の導入により、運用の安定性が向上しました。
1. 起動時間とエラー耐性の比較
数百コミット規模のリポジトリを Colab 無料枠でクローンした筆者環境での実測値です。
| 項目 / 状態 | 通常の !git clone | 改善したセットアップ | 改善効果 |
| :--------------------------- | :---------------------------: | :----------------------------------: | :------------------------- |
| 初動の転送時間 | 約 5〜10 秒（全履歴取得時） | 約 2〜4 秒（--depth 1） | 初動の短縮 |
| 同一セルの再実行 | already exists でエラー停止 | 自動消去・再取得で衝突なし | 冪等性の担保 |
| タグ push 漏れ・タイポ時 | エラーによる処理中断 | 検証を経て許可時のみ main 退避 | タグ不在時に main 代替取得 |
| 自作モジュール更新 | 再起動しないと旧コード参照 | sys.modules 破棄で即反映 | コード不整合の防止 |
2. 同一ランタイムでのセル再実行に対する冪等性と復旧
セッションが完全に切断された場合は Colab の VM ごと初期化されるため前回の残骸は残りませんが、実際の運用で頻繁に発生するのは「エラー発生後やパラメータ変更時に、同一ランタイム上でセルをもう一度実行する」ケースです。
セットアップコードの冒頭でカレントディレクトリを /content に戻し、既存フォルダを消去することで、同じセッション内でのやり直し時にディレクトリの衝突や、削除済みフォルダ内に留まって迷子になる問題を防ぎます。
※なお、pip install の実行途中で処理を強制中断した場合、site-packages 配下に不完全なパッケージが残存する可能性があります。その場合はノートブックのメニューからランタイムの再起動（Restart session）を行うか、pip install --force-reinstall を実行して修復します。
3. 今回の設計が持つトレードオフと運用上の制約
本設計は「一般利用者の手元で止まりにくく、安全に再実行できること」を優先しています。そのため、以下のトレードオフ（何を得て、何を妥協したか）が存在します。
- 【起動速度 vs Git履歴の閲覧性（--depth 1 の代償）】最新コミットに絞ることで履歴追跡は切り捨てる:
--depth 1 によって数秒での高速クローンを実現した代償として、過去のコミット履歴（git log や git diff、過去リビジョンへのチェックアウト）は利用できません。コードを配布して実行させる用途に特化し、開発者向けの履歴閲覧性は意図して捨てています
- 【可用性（止まらないこと） vs コードの再現性】main への代替切り替えは未検証コードのリスクを伴う:
タグ不在時に main ブランチへ自動退避して処理を継続させる選択肢を用意した代償として、開発途中の未検証コードが動いてしまうリスクを負います。そのため本設計では ALLOW_MAIN_FALLBACK をデフォルトで False とし、利用者がリスクを承知の上で明示的に有効化した場合のみ main へ切り替える安全弁を設けています
- 【高速な再読込 vs C拡張モジュールの更新限界】インメモリ破棄で扱えるのは純粋な Python コードに限定される:
ランタイム再起動を挟まずに sys.modules の破棄で最新コードを即時反映させる軽快さを取った代償として、numpy や torch など一度メモリに共有ライブラリ（.so）がロードされたバイナリ拡張パッケージの更新は反映できません。これらをアップグレードした場合は、OS プロセス自体の再起動（ランタイム再起動）が必要です
- 【安全なトークン注入 vs ログ出力・プロセス一覧からの漏洩リスク】例外メッセージのサニタイズが必要になる:
Secrets を用いてプライベートリポジトリを安全にクローンできる利便性を取った代償として、例外発生時にトークンが露出しないよう stderr のサニタイズ（伏字化）が必要です。また、クローン実行中は OS のプロセス一覧（ps）から一時的に引数のトークンが見える点にも留意が必要です
---
結び
Google Colab 上でリポジトリのコードを動かす作業は、ローカル開発環境での git clone とは前提が異なります。
「作業ディレクトリを /content に戻す」「既存フォルダの事前クリーンアップ」「--depth 1 による最新取得」「タグ不在時の main 代替取得」「モジュールキャッシュの破棄」という一連の対策をセットアップセルに組み込むことで、ボタン1つで安定して起動できる配布基盤を構築できます。
本稿で確立したコード展開フローと、第1弾（データ永続化編）で設計した 2 層ストレージフローを組み合わせることで、ノートブック全体のライフサイクルは以下のように整理されます。
> 【Step 1】環境セットアップ（本稿：コード展開・依存解決）
> ↓
> 【Step 2】Pull Phase（第1弾：永続層 Drive から作業層ローカルディスクへデータステージング展開）
> ↓
> 【Step 3】処理実行（ローカルディスク上での高速バッチ・クエリ処理）
> ↓
> 【Step 4】Push Phase（第1弾：成果物先行・DB末尾置換でデータステージング書き戻し ＆ アンマウント）
2 つの設計を組み合わせることで、対話的なノートブックであっても運用に耐えうる安定したバッチ実行基盤として活用できるようになります。
---
> 免責事項（Disclaimer）
> 本記事に掲載されているコードや設定例は筆者の検証環境に基づくものであり、外部サービス（Google Colab、GitHub 等）の仕様変更や実行環境の違い等によって生じたいかなる損害についても責任を負いかねます。実際の運用や設定はご自身の責任において確認の上で行ってください。
Mentions
How to Build Your Own Google Antigravity Plugin: Packaging Skills, MCP Servers, Subagents, Rules, and Hooks --- title: "How to Build Your Own Google Antigravity Plugin: Packaging Skills, MCP Servers, Subagents, Rules, and Hooks" published: true description: "A practical, step-by-step engineering guide to packaging team workflows, MCP tools, custom agents, and deterministic guardrails into a single versioned Antigravity plugin." tags: antigravity, ai, googlecloud, devtools --- > "Declarative rules define how an engineering team works; plugins make those standards portable across every repository and developer machine." When teams first adopt coding agents, customization usually starts as a single AGENTS.md file at the root of a repository. A week later, someone adds a slash-command runbook under .agents/skills/. Two weeks later, a platform engineer wires up a Model Context Protocol (MCP) server and writes a shell script to stop the agent from running terraform apply against production. In a single repository, loose files in .agents/ work fine. Across twenty microservices or a dozen infrastructure repositories, copy-pasting those files turns into configuration drift. One repo has an outdated IAM audit skill, another is missing the PreToolUse safety hook, and new hires spend their first afternoon debugging hardcoded paths in mcp_config.json. Google Antigravity Plugins solve this packaging problem. A plugin bundles Skills, MCP servers, Custom Agents (Subagents), Rules, and Lifecycle Hooks into a single versioned directory anchored by a plugin.json manifest. You can validate that directory locally with agy plugin validate --strict, share it via GitHub, drop it into a workspace, or publish it to the Antigravity Marketplace. In this guide, we will dissect how the Antigravity plugin runtime works under the hood and build a complete, production-ready plugin from scratch: gcp-terraform-kit. --- 1. Loose Workspace Files vs. Packaged Plugins Before writing our manifest, let us look at what changes when you move from ad-hoc .agents/ folders to a versioned plugin. | Dimension | Loose .agents/ Files | Antigravity Plugin (plugin.json) | | :--- | :--- | :--- | | Distribution | Manual copy-paste across repositories | Single install via agy plugin install (local, GitHub URL, or Marketplace) | | Versioning | Tracked per-repo in Git commits | Semantic versioning (MAJOR.MINOR.PATCH) in plugin.json | | Tool Namespacing | Global names; prone to collisions | Automatic namespacing (<plugin-name>_<server-name>) | | Filesystem Portability | Often relies on machine-specific paths | Built-in ${PLUGIN_ROOT} and ${PLUGIN_DATA} runtime variables | | Credential Onboarding | Manual export in shell profiles | Declarative variables prompt dialog or browser OAuth 2.0 flow | | Validation | Fails at runtime if YAML/JSON is malformed | Static verification via agy plugin validate --strict | --- 2. Anatomy of an Antigravity Plugin An Antigravity plugin is a standard directory containing a plugin.json manifest at its root (or inside .antigravity-plugin/plugin.json if you maintain a multi-editor repository alongside .claude-plugin/ or .cursor-plugin/). !Anatomy of a Google Antigravity Plugin A plugin can bundle any combination of five customization primitives: 1. Skills (skills/<skill-name>/SKILL.md): On-demand procedural runbooks, helper scripts, and reference manuals invoked automatically when a prompt matches the skill description or explicitly via /<skill-name>. 2. MCP Servers (mcp_config.json): Local stdio processes or remote Streamable HTTP endpoints that expose external tools to the agent. 3. Custom Agents & Subagents (agents/<agent-name>.md): Specialist personas with isolated context windows, dedicated model tiers (pro, flash, inherit), and restricted tool allowlists. 4. Rules (rules/AGENTS.md and rules/<rule-name>.md): Always-on architectural guardrails or file-glob-triggered coding conventions. 5. Lifecycle Hooks (hooks.json): Synchronous scripts that intercept SessionStart, PreInvocation, PreToolUse, PostToolUse, and Stop events over stdin/stdout. How Progressive Disclosure Protects Your Context Window A common mistake when building agent extensions is stuffing 40 pages of documentation into an always-on system prompt. That burns tokens on every turn and degrades model attention. !Progressive Disclosure & Lifecycle Execution Loop Antigravity avoids context bloat through progressive disclosure: - At session startup, the runtime loads rules/AGENTS.md, always_on rules, and only the name and description frontmatter of your bundled skills and subagents. - During the session, when a developer opens a .tf file, Antigravity injects rules configured with trigger: glob matching **/*.tf. - When a task matches a skill, the agent reads the full SKILL.md body and any linked files in references/ on demand. - When heavy analysis is required, the primary agent delegates to a bundled subagent in a separate conversation trajectory, keeping raw file dumps out of your main chat history. --- 3. Hands-On Tutorial: Building gcp-terraform-kit Let us build a real plugin for a Google Cloud platform engineering team. Our gcp-terraform-kit plugin will: 1. Enforce Google Cloud Terraform security conventions whenever .tf files are edited (Rules). 2. Provide an /audit-module workflow with a reference checklist (Skills). 3. Connect the official Terraform MCP server and a local state-inspector script using portable path variables (MCP Servers). 4. Ship a read-only iam-reviewer specialist agent (Subagents). 5. Block terraform apply and terraform destroy at the tool execution boundary (Lifecycle Hooks). Start by creating the directory tree: mkdir -p gcp-terraform-kit/{assets,scripts,skills/audit-module/references,agents,rules} --- Step 1: Create the Manifest (plugin.json) Every plugin requires plugin.json. Including the $schema key gives you autocomplete and inline validation in VS Code, JetBrains IDEs, and the Antigravity IDE. // gcp-terraform-kit/plugin.json { "$schema": "https://antigravity.google/schemas/v1/plugin.json", "name": "gcp-terraform-kit", "displayName": "GCP Terraform Kit", "version": "1.0.0", "description": "Google Cloud Terraform guardrails, module audit workflows, IAM reviewer subagent, and apply protection hooks.", "logo": "assets/logo.svg", "suggestedPrompts": [ "Audit the current Terraform module for CIS Google Cloud Benchmark gaps", "Review IAM bindings in this directory for least-privilege violations", "Validate and test all .tf files in the current workspace" ], "author": { "name": "Carlos Barbero", "url": "https://github.com/carlosrgomes" }, "homepage": "https://github.com/carlosrgomes/gcp-terraform-kit", "repository": "https://github.com/carlosrgomes/gcp-terraform-kit", "license": "Apache-2.0", "keywords": [ "terraform", "gcp", "google-cloud", "iac", "security" ] } > Important — Naming Rules for Strict Validation: > * name must be lowercase kebab-case (^[a-z0-9]+(-[a-z0-9]+)*$). It is used to namespace your bundled MCP servers (gcp-terraform-kit_<server>). > * displayName must not start with "Antigravity", "Gemini", "Claude", or "Cursor", and should omit suffixes like "Plugin" or "MCP". > * suggestedPrompts accepts up to 3 starter prompts that render as clickable chips on the plugin detail page in the Antigravity App. Let us also add a simple square SVG icon at assets/logo.svg (Marketplace plugins require a 1:1 square icon with ~10% padding): <!-- gcp-terraform-kit/assets/logo.svg --> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128" width="128" height="128"> <rect x="12" y="12" width="104" height="104" rx="22" fill="#0E1E40" stroke="#4CC9F0" stroke-width="4"/> <path d="M40 36L62 48V72L40 60V36ZM66 50L88 38V62L66 74V50ZM40 66L62 78V102L40 90V66Z" fill="#4CC9F0"/> </svg> --- Step 2: Author Rules (rules/AGENTS.md and Glob Rules) Plugins support two flavors of rules inside rules/: 1. rules/AGENTS.md (or rules/GEMINI.md): Loaded unconditionally whenever the plugin is enabled. No YAML frontmatter is needed. 2. Named rule files (rules/<rule-name>.md): Configured via YAML frontmatter with a trigger mode (always_on, model_decision, glob, or manual). Let us create a short rules/AGENTS.md for global plugin behavior, and put our HCL-specific rules in rules/hcl-style.md using trigger: glob so they only consume context tokens when .tf or .tfvars files are active: <!-- gcp-terraform-kit/rules/AGENTS.md --> # GCP Terraform Kit — Core Policy When working in repositories that contain Terraform configurations for Google Cloud: - Delegate IAM and security reviews to the `iam-reviewer` subagent before proposing changes to `google_*_iam_*` resources. - Never execute state-mutating commands (`terraform apply`, `terraform destroy`, `terraform state rm`) directly. Now create the glob-scoped rule in rules/hcl-style.md: --- trigger: glob globs: "**/*.tf, **/*.tfvars, **/*.tftest.hcl" description: Google Cloud Terraform coding and security standards --- # Google Cloud Terraform Standards 1. **Provider Version Pinning**: Every root module and reusable module must constrain `google >= 6.0, < 7.0` in `versions.tf`. 2. **Zero Static Keys**: Never create `google_service_account_key` resources. Use Workload Identity Federation or Service Account Impersonation. 3. **Authoritative vs. Non-Authoritative IAM**: Prefer `google_project_iam_member` over `google_project_iam_policy` or `google_project_iam_binding` to avoid wiping existing role bindings. 4. **Labels & Encryption**: Every `google_storage_bucket`, `google_sql_database_instance`, and `google_bigquery_dataset` must expose `labels = var.labels` and an optional `kms_key_name` variable for CMEK. > Warning: In rules/<rule-name>.md, the globs frontmatter field must be a comma-separated string ("**/*.tf, **/*.tfvars"), not a YAML list (["**/*.tf"]). Passing a YAML array will fail agy plugin validate --strict. --- Step 3: Package an On-Demand Skill (skills/audit-module/SKILL.md) Skills teach the agent repeatable multi-step procedures. Two rules govern plugin skills: - The name field in SKILL.md frontmatter must exactly match the parent folder name (skills/audit-module/SKILL.md -> name: audit-module). - Inside SKILL.md, reference helper scripts and docs using relative paths from the skill folder (references/cis-checklist.md). --- name: audit-module description: >- Run a structural, syntactic, and CIS Google Cloud security audit on a Terraform module. Use when the user asks to audit, review, or validate a Terraform module, or invokes /audit-module. --- # Terraform Module Audit Workflow Follow these steps in order when auditing a Terraform directory: 1. **Static Validation**: Run `terraform fmt -check -recursive` and `terraform validate` in the target directory. 2. **Checklist Verification**: Read [references/cis-checklist.md](references/cis-checklist.md) and evaluate every `.tf` file against the controls. 3. **Subagent IAM Review**: If the module declares any `google_*_iam_*` or `google_service_account` resources, delegate a deep check to the `iam-reviewer` subagent. 4. **Structured Report**: Output a markdown table with columns: `Resource`, `Severity`, `Rule Violated`, and `Recommended Fix`. Let us create the companion reference file so the skill stays concise: <!-- gcp-terraform-kit/skills/audit-module/references/cis-checklist.md --> # CIS Google Cloud Benchmark — Terraform Quick Checks - **CIS 1.4**: Ensure no `google_service_account_key` resources exist. - **CIS 5.1**: Ensure `google_storage_bucket` sets `uniform_bucket_level_access = true` and `public_access_prevention = "enforced"`. - **CIS 6.2**: Ensure `google_sql_database_instance` enables `ssl_mode = "ENCRYPTED_ONLY"` and `deletion_protection = true`. - **CIS 3.6**: Ensure `google_compute_firewall` never allows `0.0.0.0/0` on ports `22` or `3389`. --- Step 4: Configure a Working MCP Server (mcp_config.json & scripts/state-mcp.mjs) When a plugin is installed, Antigravity unpacks it into a read-only directory (~/.gemini/config/plugins/<plugin-name>) that gets replaced on upgrades. Because you never know the user's home directory path in advance, Antigravity provides three runtime variables for mcp_config.json: - ${PLUGIN_ROOT}: Resolves to the read-only installation directory of the plugin. - ${PLUGIN_DATA}: Resolves to a persistent, writable directory created specifically for your plugin outside the workspace (ideal for caches or audit logs). - ${VARIABLE_NAME}: Interpolates user-prompted secrets declared in mcpServers.<server>.variables. Let us configure a self-contained local Node.js MCP server (state-inspector) in mcp_config.json that uses ${PLUGIN_ROOT} and ${PLUGIN_DATA} so the plugin activates immediately upon installation: // gcp-terraform-kit/mcp_config.json { "mcpServers": { "state-inspector": { "command": "node", "args": [ "${PLUGIN_ROOT}/scripts/state-mcp.mjs", "--cache-dir", "${PLUGIN_DATA}/cache" ] } } } Now create the zero-dependency MCP stdio server in scripts/state-mcp.mjs. It implements the JSON-RPC 2.0 initialize, tools/list, and tools/call methods, exposing an inspect_terraform_workspace tool (namespaced at runtime as gcp-terraform-kit_state-inspector) that scans .tf files and caches the inspection report inside ${PLUGIN_DATA}: #!/usr/bin/env node // gcp-terraform-kit/scripts/state-mcp.mjs import fs from "node:fs"; import path from "node:path"; import readline from "node:readline"; const cacheDirIndex = process.argv.indexOf("--cache-dir"); const cacheDir = cacheDirIndex !== -1 && process.argv[cacheDirIndex + 1] ? process.argv[cacheDirIndex + 1] : process.env.PLUGIN_DATA || "/tmp"; function inspectWorkspace(targetDir) { const resolved = path.resolve(targetDir || "."); if (!fs.existsSync(resolved)) { return { error: `Directory not found: ${resolved}` }; } const entries = fs.readdirSync(resolved); const tfFiles = entries.filter((f) => f.endsWith(".tf") || f.endsWith(".tfvars")); const hasLockFile = entries.includes(".terraform.lock.hcl"); fs.mkdirSync(cacheDir, { recursive: true }); const result = { directory: resolved, tfFilesCount: tfFiles.length, tfFiles, hasLockFile, cacheDirUsed: cacheDir, checkedAt: new Date().toISOString() }; fs.writeFileSync( path.join(cacheDir, "last-inspection.json"), JSON.stringify(result, null, 2) ); return result; } const rl = readline.createInterface({ input: process.stdin, terminal: false }); rl.on("line", (line) => { if (!line.trim()) return; let msg; try { msg = JSON.parse(line); } catch { return; } if (msg.method === "initialize") { process.stdout.write( JSON.stringify({ jsonrpc: "2.0", id: msg.id, result: { protocolVersion: "2024-11-05", capabilities: { tools: {} }, serverInfo: { name: "state-inspector", version: "1.0.0" } } }) + "\n" ); } else if (msg.method === "tools/list") { process.stdout.write( JSON.stringify({ jsonrpc: "2.0", id: msg.id, result: { tools: [ { name: "inspect_terraform_workspace", description: "Lists .tf/.tfvars files, checks for .terraform.lock.hcl, and caches metadata in PLUGIN_DATA.", inputSchema: { type: "object", properties: { directory: { type: "string", description: "Path to the Terraform module directory." } }, required: ["directory"] } } ] } }) + "\n" ); } else if (msg.method === "tools/call") { const dir = msg.params?.arguments?.directory || "."; const report = inspectWorkspace(dir); process.stdout.write( JSON.stringify({ jsonrpc: "2.0", id: msg.id, result: { content: [{ type: "text", text: JSON.stringify(report, null, 2) }] } }) + "\n" ); } else if (msg.id !== undefined) { process.stdout.write( JSON.stringify({ jsonrpc: "2.0", id: msg.id, result: {} }) + "\n" ); } }); > Tip — Prompting Users for API Keys (variables): If your plugin connects to a remote MCP endpoint that requires an API token, declare a variables array inside that server entry in mcp_config.json: > ```json > "variables": [ > { > "name": "TFE_TOKEN", > "title": "Terraform Registry API Token", > "description": "Team or user API token with read access to private modules.", > "link": "app.terraform.io" > } > ] > ``` > When a server declares variables, agy plugin install stages the plugin in a disabled state until the user opens /plugin (or Customizations > Installed in the App) to fill in the token dialog. --- Step 5: Define a Specialist Subagent (agents/iam-reviewer.md) Custom agents live in agents/<agent-name>.md. By setting both subagent: true and mainAgent: true, this specialist can either be spawned automatically in the background by the primary agent or selected directly by the developer via agy --agent iam-reviewer (or /agents inside the TUI). Notice how we omit edit_file, write_file, and run_command from the tools list to guarantee the reviewer operates strictly in read-only mode: --- name: iam-reviewer description: >- Specialist auditor for Google Cloud IAM bindings, Service Accounts, and Workload Identity Federation in Terraform. Delegate to this agent whenever reviewing or modifying IAM permissions. model: pro mainAgent: true subagent: true commandExecutionPolicy: sandbox tools: - view_file - grep_search - find_by_name - list_dir - send_message inheritMcp: true --- You are a Principal Google Cloud Security Architect specializing in IAM and Terraform. Your job is to inspect `.tf` files for privilege escalation paths, primitive roles (`roles/owner`, `roles/editor`), static keys (`google_service_account_key`), and authoritative bindings (`google_project_iam_policy`) that risk production outages. For every issue found, return: 1. File path and line range. 2. The blast radius of the current configuration. 3. The exact drop-in HCL replacement using `google_project_iam_member` or Workload Identity. --- Step 6: Enforce Deterministic Gates with Lifecycle Hooks (hooks.json) Rules and system prompts guide the model's reasoning, but safety-critical boundaries belong in Lifecycle Hooks. Hooks execute synchronous shell commands when lifecycle events fire. In a plugin, hooks.json must sit at <plugin-root>/hooks.json. Hook commands always execute with their working directory (cwd) set to <plugin-root>, so you should reference bundled scripts using relative paths (./scripts/block-apply.mjs) rather than ${PLUGIN_ROOT}. // gcp-terraform-kit/hooks.json { "terraform-safety-gate": { "enabled": true, "PreToolUse": [ { "matcher": "run_command", "hooks": [ { "type": "command", "command": "node ./scripts/block-apply.mjs", "timeout": 5 } ] } ] } } Now let us write scripts/block-apply.mjs. Hook scripts read the event payload as JSON from stdin (using camelCase keys like toolCall.args.CommandLine) and write a JSON decision object to stdout with exit code 0: #!/usr/bin/env node // gcp-terraform-kit/scripts/block-apply.mjs import fs from "node:fs"; const rawInput = fs.readFileSync(0, "utf-8"); const payload = JSON.parse(rawInput || "{}"); const cmd = payload?.toolCall?.args?.CommandLine || ""; const forbiddenPattern = /\bterraform\s+(apply|destroy|state\s+rm|force-unlock)\b/i; if (forbiddenPattern.test(cmd)) { process.stdout.write( JSON.stringify({ decision: "deny", reason: "Blocked by gcp-terraform-kit: state-mutating Terraform commands (apply, destroy, state rm, force-unlock) must run via CI/CD pipelines, not directly from the agent." }) ); process.exit(0); } process.stdout.write(JSON.stringify({ decision: "allow" })); Add a concise README.md at the plugin root for the Marketplace detail view, and mark both scripts in scripts/ as executable (0755) so Marketplace bundles preserve execution permissions: cat << 'EOF' > gcp-terraform-kit/README.md # GCP Terraform Kit Google Cloud Terraform guardrails, module audit workflows, IAM reviewer subagent, and apply protection hooks for Google Antigravity. EOF chmod 0755 gcp-terraform-kit/scripts/*.mjs --- 4. Validating, Testing, and Installing Your Plugin Before installing the plugin into Antigravity, we can smoke-test our PreToolUse hook and our state-inspector MCP server directly from the terminal: # Test 1: Verify the hook blocks `terraform apply` node ./gcp-terraform-kit/scripts/block-apply.mjs <<< '{"toolCall":{"name":"run_command","args":{"CommandLine":"terraform apply -auto-approve"}}}' # Output: {"decision":"deny","reason":"Blocked by gcp-terraform-kit: state-mutating Terraform commands..."} # Test 2: Verify the hook allows `terraform validate` node ./gcp-terraform-kit/scripts/block-apply.mjs <<< '{"toolCall":{"name":"run_command","args":{"CommandLine":"terraform validate"}}}' # Output: {"decision":"allow"} # Test 3: Verify the MCP server responds to `initialize` and `tools/list` printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}\n{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}\n' \ | node ./gcp-terraform-kit/scripts/state-mcp.mjs --cache-dir /tmp/tf-kit-cache Now run the built-in validator from the Antigravity CLI (agy). Passing --strict treats warnings as errors and checks your plugin.json schema, skill directory naming, rule glob syntax, hook paths, and logo dimensions: ![Testing Locally with agy plugin validate --strict] !Image description # 1. Run strict validation against your plugin folder agy plugin validate --strict ./gcp-terraform-kit # Validating gcp-terraform-kit (.../gcp-terraform-kit/plugin.json) # skills 1 · agents 1 · rules 2 · hooks 1 · mcpServers 1 · sidecars 0 # ✔ Validation passed # 2. Install the plugin into ~/.gemini/config/plugins/gcp-terraform-kit agy plugin install ./gcp-terraform-kit # [ok] gcp-terraform-kit # ✔ skills : 1 installed (audit-module) # ✔ agents : 1 installed (iam-reviewer) # ✔ rules : 2 installed (AGENTS, hcl-style) # ✔ mcpServers : 1 installed (state-inspector) # ✔ hooks : 1 installed (terraform-safety-gate) # 3. Verify the plugin is enabled and its subagent is registered agy plugin list agy agents Four Common Pitfalls Caught by --strict If agy plugin validate --strict reports an error, check these four frequent mistakes first: 1. YAML Array in Rule globs: Writing globs: ["*.tf", "*.tfvars"] instead of a comma-separated string globs: "**/*.tf, **/*.tfvars". 2. Using ${PLUGIN_ROOT} in hooks.json: ${PLUGIN_ROOT} is expanded in mcp_config.json, not in hooks.json. Because hooks already run with cwd set to <plugin-root>, always use relative paths like node ./scripts/block-apply.mjs. 3. Skill Directory Mismatch: Naming the directory skills/tf-audit/ while setting name: audit-module inside SKILL.md. Both must match (skills/audit-module/SKILL.md). 4. Root-Level variables in mcp_config.json: Placing the variables array at the root of mcp_config.json instead of nesting it inside the specific server (mcpServers.<server-name>.variables). --- 5. Distributing Your Plugin to Your Team Once your plugin passes strict validation, you have three ways to distribute it depending on your team's workflow: Option A: Commit to a Repository (.agents/plugins/) For repository-specific plugins that should activate automatically when any developer clones the repo, place the folder under .agents/plugins/gcp-terraform-kit/. If your organization keeps shared plugins in a central monorepo directory (for example, platform/agent-plugins/), register that directory in .agents/plugins.json: // .agents/plugins.json { "entries": [ { "path": "platform/agent-plugins", "exclude": ["experimental-.*"] } ] } Option B: Install Directly from GitHub Push gcp-terraform-kit to a GitHub repository. Any engineer on your team can install it across all their workspaces with a single command: agy plugin install https://github.com/your-org/gcp-terraform-kit Option C: Publish to the Antigravity Marketplace If you are building an integration for an open-source tool, cloud platform, or developer product, add a README.md at the plugin root documenting prerequisites and example workflows, verify that agy plugin validate --strict exits with 0 errors and 0 warnings, and submit your repository through the official Google Antigravity Plugins documentation. --- Wrapping Up Moving from scattered prompt files to a structured Antigravity plugin takes less than fifteen minutes, and it changes how your team maintains AI workflows: - Skills give engineers repeatable slash-command runbooks (/audit-module) without bloating the context window at startup. - MCP servers wire up live schemas and internal APIs using portable ${PLUGIN_ROOT} and ${PLUGIN_DATA} variables. - Subagents isolate heavy security and architecture reviews into dedicated read-only personas. - Rules and Lifecycle Hooks pair context-aware guidance with deterministic PreToolUse safety gates. Pick one workflow your team repeats every week—whether that is auditing Terraform modules, reviewing Firestore security rules, or verifying Kubernetes manifests—package it into a plugin.json bundle, and run agy plugin validate --strict.
Give Claude Code and Cursor a shared, auto-updating map of your codebase AI coding assistants are great at writing code and surprisingly bad at knowing where things are. Ask "what breaks if I change this function?" on a large repository and the assistant starts opening files one by one, burning tokens and context, and often still misses a caller three modules away. Graphify fixes that by turning a codebase into a knowledge graph: functions, classes, modules and the edges between them, parsed locally with tree-sitter. Your assistant queries the graph through MCP instead of re-reading the code. With over 130k GitHub stars, it clearly hit a nerve. This post shows how to run Graphify as one shared MCP server that every assistant on your team can use, kept up to date automatically. Everything here is open source. Local graph vs. shared server The default Graphify workflow is local: you run it in your repo, it writes graphify-out/graph.json, and your assistant reads it. If you work alone on one machine, that's perfect. Committing graphify-out/ and installing Graphify's git hook even keeps several of your own machines in sync. It gets clumsier when: - several people (or several agents) need the same graph, - you don't want everyone to install and run Graphify locally, - you want the graph to follow main without anyone thinking about it. Graphify already ships an HTTP transport for its MCP server, so the missing piece is just the plumbing: clone, build, serve, refresh, secure. That's what graphify-railway does. How it works your Git repo ──clone/pull──▶ graphify extract --code-only ──▶ graph.json (volume) │ reloaded on change Claude Code / Cursor ──HTTPS + API key──▶ /mcp ◀── graphify.serve --transport http 1. Clone. On start, the service clones REPO_URL. Private repos work with a read-only token. 2. Build. It runs graphify extract --code-only: pure local AST parsing, no LLM API key. On a small repo (about 100 nodes) it takes under a second. 3. Serve. It starts graphify.serve over MCP Streamable HTTP on /mcp. Every request must carry the API key. 4. Refresh. Every hour (configurable), it pulls the repo. If the commit changed, it rebuilds and swaps graph.json atomically. Graphify's server checks the file on each request and reloads it, so connected clients keep their session: no restart, no downtime. A few details I cared about: - The git token never touches the disk. It's passed to each git command as an HTTP header, not written to .git/config, and it's redacted from logs. - The API key isn't on the command line. It goes through the environment, so it doesn't show up in ps. - The last good graph survives. It lives on a persistent volume: if the repo is briefly unreachable at startup, the previous graph is served; if a refresh fails, the old graph stays online. - Graphify is pinned to a tested release in the Dockerfile. Graphify ships very often, and you don't want a surprise upgrade on a team server. The whole wrapper is one Python file and a Dockerfile. Deploy it One click on Railway Deploy on Railway. You get the service and a volume. Fill in one variable, REPO_URL (for example https://github.com/your-org/your-repo); the API key is generated for you. For a private repo, also set GIT_TOKEN. On GitHub, use a fine-grained token limited to that repository with Contents: Read-only. Anywhere you can run a container docker build -t graphify-server https://github.com/jean82290/graphify-railway.git docker run -d -p 8080:8080 \ -v graphify-data:/data \ -e REPO_URL=https://github.com/your-org/your-repo \ -e GRAPHIFY_API_KEY="$(openssl rand -hex 24)" \ graphify-server Put it behind HTTPS before sharing it with anyone. | Variable | Required | Default | What it does | |---|---|---|---| | REPO_URL | yes | | HTTPS URL of the repository to map | | GRAPHIFY_API_KEY | yes | | Key every MCP client must send | | GIT_TOKEN | private repos | | Read-only access token | | REPO_BRANCH | no | default branch | Branch to map | | REFRESH_MINUTES | no | 60 | How often to pull and rebuild (0 disables) | Connect your assistant Claude Code: claude mcp add --transport http graphify https://YOUR-SERVICE.up.railway.app/mcp \ --header "Authorization: Bearer YOUR_GRAPHIFY_API_KEY" Cursor (.cursor/mcp.json): { "mcpServers": { "graphify": { "url": "https://YOUR-SERVICE.up.railway.app/mcp", "headers": { "Authorization": "Bearer YOUR_GRAPHIFY_API_KEY" } } } } Any MCP client that supports Streamable HTTP with custom headers works the same way. What your assistant can do with it Graphify exposes tools such as query_graph, get_node, get_neighbors, get_community, god_nodes, graph_stats and shortest_path, plus PR-oriented tools (list_prs, get_pr_impact, triage_prs). In practice, this means questions like these get answered from the graph instead of a file-by-file crawl: - "What calls buildEvent, directly or indirectly?" - "What's the shortest path between the HTTP handler and the database layer?" - "Which parts of the code are the most connected, and therefore the riskiest to change?" When I tested it on a small Node.js project, a query_graph call starting from one function returned its 80 related nodes and 152 edges in a single response. That's the context an assistant would otherwise have to rebuild by opening a dozen files. Security notes - The server is reachable on the internet, and the API key is what keeps your code map private. Rotate it by changing GRAPHIFY_API_KEY and redeploying. - The graph contains names and relationships from your code, not the full source, but treat it as confidential if your code is. - Use a token with read access to a single repository. Limitations - Code only, for now. Graphify can also extract docs, PDFs and images, but that pass needs an LLM backend and an API key. The wrapper sticks to the free, offline AST mode. - One repository per instance. If you need several, run several instances (or tell me you'd like multi-repo support). - Hourly refresh by default. Lower REFRESH_MINUTES if you need fresher graphs. Links - Source (MIT): jean82290/graphify-railway - One-click deploy: Railway template - Graphify itself (Apache-2.0): Graphify-Labs/graphify Disclosure: this is an unofficial wrapper, not affiliated with Graphify Labs. If you deploy through the Railway template, I receive a share of Railway's hosting fees. The Docker route above costs me nothing and works just as well. Questions, ideas or bugs: open an issue on GitHub or leave a comment below. I'm especially curious whether multi-repo support would be useful to you.
Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins How the FiftyOne plugin framework turns your MCAP and LeRobot curation checks into buttons, panels, and background jobs that run on every episode, and why that works across a whole dataset rather than one recording. You have 400 teleoperation episodes and a policy to train by Friday. Some of those episodes are bad. The operator hesitated for eight seconds before the grasp. The wrist camera froze halfway through. The gripper opened and closed three times because the first grasp slipped. The task string says "pick up" and nothing else. A policy will learn from all of it. You could open each episode in a log viewer and scrub through it. At two minutes per episode, that's more than 13 hours, and you'll stop paying attention long before the end. What you want is a ranked list, worst first, with every flag one click away from the moment it happened. No viewer ships that list, because the checks depend on your robot. A dual-arm rig with multi-joint hands needs different smoothness metrics than a wheeled rover with an inertial measurement unit (IMU) and GPS. The tool that curates your data has to be one you can extend in Python, on your schema, across the whole dataset. That's what the FiftyOne plugin framework is for. Key takeaways - A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job. - FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code. - FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution. - Most viewer extensions run in the browser against one recording, and labeling-platform hooks act on one workflow task. FiftyOne plugins run server-side Python with custom UI across a whole dataset. - Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings. Why physical AI data needs custom tooling Image datasets are forgiving. Every sample is a picture, every label is a box or a mask, and the same quality checks work on most of them. Robot data has no such standard shape. One MCAP file from an autonomous vehicle might hold six cameras, two LIDARs, an IMU, GPS fixes, and a /tf tree. One LeRobot episode from a tabletop arm holds two camera streams, a 14-dimensional observation.state, a matching action array, and a task string. An egocentric dataset adds audio and hand poses. The questions you need answered are specific to that shape: - Is the motion in this episode smooth, or full of corrections? - Did any sensor drop frames, drift its clock, or clip its values? - Does the action track the state, or is there lag between video and action? - Is the gripper behavior consistent with the other episodes of the same task? - Is the language instruction usable for fine-tuning a vision-language-action (VLA) model? A generic viewer can't answer those. It can only show you the data and wait for you to notice. FiftyOne gives you the dataset layer: episodes as samples, fields you can filter and sort on, views, tags, and an App with a multimodal viewer that plays MCAP and LeRobot streams in lockstep. Plugins give you the part that's specific to your robot. !Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation. Diagram of FiftyOne at the center of an ML stack, with plugin operators and panels connecting it to labeling tools, vector search backends, model inference, and automation. What is a FiftyOne plugin? A plugin is a folder FiftyOne finds. If the folder contains a fiftyone.yml, FiftyOne loads it at runtime and adds whatever it declares to the App, the SDK, and the CLI. Plugins add two kinds of things: - Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator. - Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel. Under the hood, a panel is a special kind of operator, so everything below about operators applies to panels too. | | Notebook script | FiftyOne plugin operator | | --- | --- | --- | | Who can run it | You, in your kernel | Anyone with the App open, no code | | Inputs | Variables you edit by hand | A typed form, validated before anything runs | | What it acts on | The dataset you loaded | The view or selection the user has open | | Long jobs | Your kernel is busy until it finishes | Delegated to a worker, so the App stays free | | Reuse | Copy and paste | One URI: buttons, panels, pipelines, notebooks, agents | What changes when you promote a notebook script to a FiftyOne plugin operator. Good to know. Why not keep it in a notebook? Start in a notebook. Promote the code to an operator when someone else needs to run it, or when you catch yourself running it a second time. The figure below shows what changes. !Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse. Comparison of a notebook script and a FiftyOne plugin operator across who can run it, inputs, what it acts on, long jobs, and reuse. The row that matters most for robot data is "what it acts on". A script scores whatever dataset you loaded. An operator scores the view the user has open right now, so the same code works on one task, one robot, one week of collection, or the episodes someone tagged for review. The anatomy of a FiftyOne plugin Here's the folder for a plugin with a Python backend and a custom React panel: !FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App. FiftyOne plugin folder with required fiftyone.yml and _init_.py files and optional requirements.txt and JavaScript files, showing what loads on the Python server and what loads in the App. - fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs. - __init__.py holds your classes and a register(p) function. - requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends. - Everything below the dashed line is optional JavaScript for custom UI. A Python-only plugin is two files. Teal in every figure in this post means the FiftyOne server, which runs Python. Orange means the App in your browser. Keep it honest. Every operator name listed in fiftyone.yml must match the name in that operator's config. If you register a class but forget to list it, the operator doesn't show up, and you get no error. It's the most common "where did my operator go" bug. A complete operator in 25 lines This operator tags samples whose detections fall below a confidence threshold: import fiftyone as fo import fiftyone.operators as foo import fiftyone.operators.types as types from fiftyone import ViewField as F class TagLowConf(foo.Operator): @property def config(self): return foo.OperatorConfig(name="tag_low_conf", label="Tag low-confidence") def resolve_placement(self, ctx): return types.Placement( types.Places.SAMPLES_GRID_SECONDARY_ACTIONS, types.Button(label="Tag low-conf"), ) def resolve_input(self, ctx): inputs = types.Object() fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections)) inputs.enum("field", fields, required=True, label="Label field") inputs.float("thresh", default=0.3, label="Confidence threshold") return types.Property(inputs) def execute(self, ctx): field, thresh = ctx.params["field"], ctx.params["thresh"] view = ctx.target_view().filter_labels(field, F("confidence") < thresh) view.tag_samples("low_conf") ctx.ops.notify(f"Tagged {len(view)} samples") def register(p): p.register(TagLowConf) And here's what the user sees: !TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App. TagLowConf operator code color-coded to the button, input form, and notification it creates in the FiftyOne App. Four methods, four jobs: 1. config names the operator. name is its ID, and label becomes the form's heading. 1. resolve_placement puts a "Tag low-conf" button in the grid's action bar. 1. resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines. 1. execute does the work on the target view and sends a notification. You wrote no UI code. You described the inputs and the App built the interface. Swap the body of execute for spectral arc length (SPARC) smoothness over an /odom channel, and you have the skeleton of an MCAP quality scorer. How an operator call works Every operator call is two round trips between the App and the FiftyOne server: !Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations. Sequence diagram of a FiftyOne operator call, with one round trip that resolves inputs into a form and a second that executes and returns UI operations. 1. Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel. 1. Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them. Your code runs only on the server. The browser draws what the server tells it to. That's why an operator can decode protobuf MCAP messages with mcap-protobuf-support, run SciPy's Welch power spectral density (PSD) on an IMU stream, or call a model on a GPU, without shipping any of that to the browser. When something breaks, ask which arrow failed. A wrong form means resolve_input. A fine form that does nothing means execute. An execute that ran but didn't change the App means the UI operations you returned. Long jobs run in the background with delegated execution Scoring 400 episodes is not a click-and-wait job. Decoding video alone costs lerobot_data_curation about 0.4 seconds per camera per episode. !Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them. Flowchart comparing immediate execution in the FiftyOne server with delegated execution, where jobs queue in MongoDB and an orchestrator runs them. Set allow_delegated_execution=True in the operator's config, and the user can choose. Immediate execution runs inside the server while the App waits, with a live progress bar. Delegated execution writes the job to a queue in MongoDB, and a separate worker runs the same execute method. You start a worker with fiftyone delegated launch and track jobs with fiftyone delegated list. Both physical AI plugins below default to delegated execution, because a real corpus takes minutes. Keep it honest. If no worker is running, delegated jobs sit in the queue forever. And a delegated job has no browser attached, so write results to the dataset rather than trying to change the user's view. Panels: UI that stays open A curation tool needs more than a form. It needs a place to see the ranking, click into an episode, and tag it. That's a panel. !FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields. FiftyOne panel lifecycle from on_load to render to event handlers, next to four places to keep state: panel state, panel data, execution store, and dataset fields. A panel's on_load runs when it opens, render returns a layout, and events call your Python handlers with a fresh context. The event that matters most for curation is on_change_view: when the user filters the grid, your panel re-computes for the new view. Here's a complete panel that shows class counts and follows the grid: class ClassCounts(foo.Panel): @property def config(self): return foo.PanelConfig(name="class_counts", label="Class counts") def on_load(self, ctx): self.refresh(ctx) def on_change_view(self, ctx): self.refresh(ctx) def refresh(self, ctx): counts = ctx.view.count_values("ground_truth.detections.label") ctx.panel.set_state("n", len(counts)) ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}]) def render(self, ctx): panel = types.Object() panel.md(f"**{ctx.panel.get_state('n')} classes**") panel.plot("counts") return types.Property(panel) Small values go in panel state, which your Python handlers can read. Large payloads like chart data go through set_data, which is write-only from Python. Settings you want to survive a reload go in the execution store, ctx.store. Results that belong to the data go in dataset fields, where you can filter and sort on them like any other field. When the built-in widgets run out, you have two options. A hybrid panel keeps state and compute in Python and renders a React component you write. A JS panel puts React in charge of the UI and calls your Python operators whenever it needs the server. Both plugins below are JS panels backed by Python operators, which suits dense, interactive dashboards. If you build a React UI, VOODO, the component library the FiftyOne App itself uses, makes it look native. !Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build. Comparison of Python, hybrid, and JS FiftyOne panels showing what runs in the browser, what runs on the server, and which need a Node build. Operators compose Every operator has a URI, like @voxel51/brain/compute_similarity, and a typed input schema. Anything that can call an operator can call yours. !One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates. One FiftyOne operator called from eight places: an App button, the operator browser, another operator, a pipeline stage, a notebook, AI agents through the MCP server, a background worker, and teammates. That includes the FiftyOne Model Context Protocol (MCP) server, which exposes list_operators, get_operator_schema, and execute_operator to AI agents. An agent can find your quality scorer, read its inputs, run it on the episodes tagged review, and report back which ones failed sensor health. The schema validates the agent's JSON the same way it validates a person filling in the form. For a physical AI team, that means the curation logic you write once runs from a button during review, from a notebook in your training pipeline, from a background worker overnight, and from an agent. How FiftyOne plugins differ from viewer extensions Most robotics viewers have extension points, and they're usually scoped to the recording you have open: a custom panel, a message converter, or a new file loader that runs in the browser or desktop app. Labeling platforms add hooks that act on one task in a workflow stage. FiftyOne plugins run on the server, in Python, against any view of a dataset. A single plugin can add a form, run a job across every episode in the background, render a custom panel with the results, and expose all of it to agents through the MCP server, without leaving the language your robotics stack is written in. For live debugging of a single recording, a dedicated log viewer is the right tool, and FiftyOne doesn't try to replace it. FiftyOne reads .rrd files and MCAP files, so a common setup is to debug individual logs in a viewer and curate the dataset in FiftyOne. Two FiftyOne plugins for MCAP and LeRobot curation Both plugins below were built using only the pieces covered above. Both follow the same rule, stated in their READMEs: the scores order your review queue, and a person makes the call. A perfectly smooth demonstration can still show the wrong task. LeRobot Data Curation lerobot_data_curation ranks the episodes of a LeRobot v3 dataset worst-first, so you know which to look at before training a policy or fine-tuning a VLA. It scores each episode on motion smoothness, time efficiency, action-state tracking, gripper behavior, consistency, integrity, and language, and optionally on camera quality: blur, exposure, clipping, frozen feeds, and video-action lag. It has no model-based metrics, so there's no model to download. How it maps to the framework: - A typed form with tabs. The compute operator's form has Data, Metrics, Camera, and Normalization tabs. You pick the observation.state and action arrays, and assign arm and gripper joints by name. Nothing is guessed from the data, and your picks are remembered for the next run. - Delegated by default. The operator sets allow_delegated_execution=True and default_choice_to_delegated=True, so scoring runs in the background. Camera decoding is the slow part. - A JS panel backed by Python operators. The LeRobot Curation panel is a React component with Overview, Motion & Action, Integrity & Coverage, Vision and Language tabs. Click a histogram bar to filter the grid to those episodes. Click a row to open an inspector with joint traces, the speed profile, the gripper timeline, and frames picked around the flagged spans. - It follows the view. The panel refreshes when the view changes, so filtering the grid re-ranks it. - Tags on the timeline. Flagged spans such as idle stretches, the longest pause, and acceleration spikes are written as temporal tags on each episode so that you can scrub straight to them in the multimodal viewer. Footer buttons apply review, exclude-candidate, and relabel sample tags. import fiftyone as fo dataset = fo.Dataset.from_dir( dataset_dir="/data/lerobot/my_dataset", dataset_type=fo.types.LeRobotDataset, ) session = fo.launch_app(dataset) # then run "LeRobot curation: compute quality" from the operator browser Keep it honest. Scores are relative to the batch you scored. Per-task normalization needs about 20 episodes per task, and the default thresholds were calibrated on a 102-episode development set. Check them on your own data. Cloud-hosted LeRobot sources aren't supported yet. Demo Quality Scorer demo_quality_scorer does the same job for multimodal MCAP datasets: teleoperations, autonomous vehicles, drones, rovers, and egocentric capture. !Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer. Demo Quality Scorer plugin in the FiftyOne App ranking robot episodes from a multimodal MCAP dataset and opening one in the multimodal viewer. It computes three metric families and ranks episodes worst-first: 1. Motion smoothness: SPARC, log dimensionless jerk (LDLJ), jerk RMS, and a low-to-high frequency power ratio, computed over windows of a speed profile you choose from position or velocity signals such as /odom -> twist_linear. 1. Sensor health: dropout, rate stability, clock drift, clipping, and cross-channel desync, all from message timestamps and raw values, so it works on any channel the plugin can decode. 1. Outliers: models that flag episodes whose motion and health features don't look like the rest of the batch. How it maps to the framework: - A dynamic form. The form inspects the first sample's channels. If no telemetry channel carries a numeric signal, the motion family switches itself off and says why. If the server lacks a decoder, the form names the package to install. - Immediate or delegated, your choice. Run immediately with an in-app progress bar, or schedule it in the background (default). - A JS panel. The Episode Quality panel is a React component that has Motion, Health, and Outliers tabs. Click a histogram bar to filter the grid. Click a table row to open that episode in the multimodal viewer, with a toast naming the timecode of its worst flagged interval. - Results as fields. Scores land in quality.* fields such as quality.sparc and quality.ldlj, so you can sort and filter on them outside the panel too. pip install "fiftyone[multimodal]>=1.22.0" mcap mcap-protobuf-support \ mcap-ros1-support mcap-ros2-support numpy scipy scikit-learn fiftyone plugins download https://github.com/harpreetsahota204/demo_quality_scorer Keep it honest. Structural errors, such as the wrong action at a key moment, are invisible to every metric here. Re-running over a different set of episodes re-fits normalization, so expect scores to move. Build your own in six steps Your robot has checks neither plugin covers. Here's the loop for writing them. !Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write. Six steps to build a FiftyOne plugin: scaffold, write, register, run, iterate, and ship, with a loop from iterate back to write. Step 1: Scaffold the plugin Run fiftyone plugins create or make a folder in your plugins directory with a fiftyone.yml: name: "@your-org/gripper_checks" description: Flags episodes with repeated grasp attempts operators: - score_grasps Step 2: Write the operator One class per operator or panel, like the examples above. Keep the scoring logic in a plain Python function so a notebook, a test, and the operator can all call it. Step 3: Register it Call p.register(ScoreGrasps) inside register(p), and make sure the name matches fiftyone.yml. Step 4: Run the App in debug mode export VFF_MULTIMODAL=1 fiftyone app debug my_lerobot_dataset fiftyone app debug prints server logs and tracebacks to your terminal. VFF_MULTIMODAL=1 turns on the multimodal viewer in the App; set it before fiftyone is imported in every process that touches a multimodal dataset. Step 5: Iterate Edit, refresh the App, repeat. Python changes are picked up on the next request without a server restart. If you have a React panel, keep npm run dev rebuilding it. Step 6: Ship it Push to GitHub. Anyone installs it with: fiftyone plugins download https://github.com/your-org/gripper_checks fiftyone plugins requirements @your-org/gripper_checks --install Good to know. How do I test an operator without the App? Operators are callable from Python with foo.execute_operator, so you can test them headless in pytest. Try it yourself - Plugin developer guide - FiftyOne multimodal user guide for MCAP and LeRobot - voxel51/fiftyone-plugins, the official plugins and panel examples - lerobot_data_curation - demo_quality_scorer - FiftyOne MCP server - VOODO, the React design system for plugin panels Frequently Asked Questions How do I curate a LeRobot dataset before training a policy? Import it with fo.Dataset.from_dir(dataset_dir=..., dataset_type=fo.types.LeRobotDataset), where each sample is one episode. Then install lerobot_data_curation and run its compute operator to rank episodes worst-first on motion, gripper, camera, and language metrics. Review the top of the list and tag episodes as exclude-candidate or relabel. Can FiftyOne open MCAP files? Yes. FiftyOne treats .mcap, .bag, and .rrd files as media_type="multimodal" samples, one episode per file, and plays their streams in a synchronized multimodal viewer. Set VFF_MULTIMODAL=1 before importing fiftyone to turn the viewer on. How do I find bad demonstrations in robot teleoperation data? Score every episode on motion smoothness (SPARC, LDLJ, jerk) and sensor health (dropout, rate stability, clock drift), then sort worst-first. demo_quality_scorer does this for MCAP datasets. Treat the scores as a review queue, because a smooth demonstration can still show the wrong task. Do FiftyOne plugins need JavaScript? No. A Python-only plugin is fiftyone.yml plus __init__.py, and the App builds forms and panels from Python declarations. Add React only when you need custom interactivity, such as the dashboards in lerobot_data_curation and demo_quality_scorer. Where does FiftyOne plugin code run? It runs on the FiftyOne server, in Python, or on a background worker for delegated operations. It never runs in the browser, so it can use any Python library, a GPU, or secrets without exposing them.
I built a free validated n8n template library with 54 workflows (CI-tested, documented) — feedback welcome # Release v0.2.0 — OpenAI Ecosystem Focus + First Validated Workflows ## What's new ### ✅ First ready-status workflows Two workflows are now fully validated with complete setup documentation, sample payloads, SQL scripts, and cost tables: - [Translate Form Submissions](AI_ML/translate_form_submissions.json) — GPT-4o + text-embedding-3-small + Supabase pgvector Translates multilingual webhook form payloads and logs results to Google Sheets. - [Daily Content Ideas](AI_ML/daily_content_ideas.json) — GPT-4o + Cohere embeddings + Supabase deduplication Generates and deduplicates daily content ideas, posts to Slack. ### 📋 OpenAI workflow index The root README now has a dedicated section listing all 10 OpenAI-powered workflows with their specific models (gpt-4o, text-embedding-3-small, whisper-1). ### 🔧 Stronger CI - JSON schema validation added to scripts/validate_workflows.py via optional jsonschema dependency - scripts/requirements.txt makes CI dependencies reproducible - CI installs dependencies before running validator ### 📖 Richer docs for validated workflows Validated workflow docs now include: - Full node inventory table - Sample input JSON payloads - Sample output format - Step-by-step credential setup - Supabase pgvector SQL - Per-provider cost estimates ## Install / use ```bash # Clone and validate git clone your-org/n8n-free-templates cd n8n-free-templates pip install -r scripts/requirements.txt python scripts/validate_workflows.py ``` Import any .json file into n8n via Settings → Import from File. ## What's next (v0.3.0) - Validate remaining AI/ML workflows - Add category READMEs with compatibility notes - Add error handling and retry guidance to workflow docs
I built a free validated n8n template library with 54 workflows (CI-tested, documented) — feedback welcome # Release v0.2.0 — OpenAI Ecosystem Focus + First Validated Workflows ## What's new ### ✅ First ready-status workflows Two workflows are now fully validated with complete setup documentation, sample payloads, SQL scripts, and cost tables: - [Translate Form Submissions](AI_ML/translate_form_submissions.json) — GPT-4o + text-embedding-3-small + Supabase pgvector Translates multilingual webhook form payloads and logs results to Google Sheets. - [Daily Content Ideas](AI_ML/daily_content_ideas.json) — GPT-4o + Cohere embeddings + Supabase deduplication Generates and deduplicates daily content ideas, posts to Slack. ### 📋 OpenAI workflow index The root README now has a dedicated section listing all 10 OpenAI-powered workflows with their specific models (gpt-4o, text-embedding-3-small, whisper-1). ### 🔧 Stronger CI - JSON schema validation added to scripts/validate_workflows.py via optional jsonschema dependency - scripts/requirements.txt makes CI dependencies reproducible - CI installs dependencies before running validator ### 📖 Richer docs for validated workflows Validated workflow docs now include: - Full node inventory table - Sample input JSON payloads - Sample output format - Step-by-step credential setup - Supabase pgvector SQL - Per-provider cost estimates ## Install / use ```bash # Clone and validate git clone your-org/n8n-free-templates cd n8n-free-templates pip install -r scripts/requirements.txt python scripts/validate_workflows.py ``` Import any .json file into n8n via Settings → Import from File. ## What's next (v0.3.0) - Validate remaining AI/ML workflows - Add category READMEs with compatibility notes - Add error handling and retry guidance to workflow docs
Claude Took Anthropic From 2,500 to 30,000 Merges a Day. Then It Moved Off Git - Book: AI Agents Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- In May 2026, Anthropic merged around 2,500 code changes a day. By September it was merging more than 30,000 a day, according to the company's engineering blog. Anthropic names the cause itself. Claude's ability to write code, the post says, is "what put us in this situation in the first place." It also names where things gave way first: "The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI." On October 13, Anthropic published what it did about it. A team of seven built Scion, an in-house version control system, plus a code review tool called Antique, and fully migrated the monorepo off git in five months. The post drew a busy Hacker News thread within hours. What broke at 30,000 merges a day The scale numbers in Anthropic's write-up explain why. "The history of our main branch alone is almost 100 GB, and a compressed working copy is over 25 GB." Anthropic runs "on the order of 25 million code checkouts per day for CI." Then the line that ends the argument: "Fetched naively, that would be over 600 PB of git traffic per day." Anthropic's diagnosis is short: "Git assumes that clients work offline and need a complete copy of the repository." That design is why git is pleasant on a laptop. It is also why every CI job that wants one directory still has to negotiate with history it will never read. Anthropic had been absorbing that load with a fleet of git mirrors, which it now calls "our previous git mirroring infrastructure." !A classic branching version-control graph drawn as a tree, overwhelmed by a dense swarm of small orange commit nodes pouring in from robotic hands, with a narrow pipe at the base straining to carry them When agents write the code, the repository's plumbing becomes the slow part. How Scion breaks git's assumptions Scion starts from the opposite premise, which Anthropic states in four words: "Clients are always online." The rest of the design follows from that: - The server owns history, blame, commits, merges and rebases. The client keeps a bounded local cache and the working files. - Checkouts are sparse by default. Scion's protocol "is centered around sparse checkouts, which only contain the directories the client asks for." Some jobs skip the checkout entirely: "a CI job that needs the diff between a commit and its merge base, for example, can ask the server for it." - Fetches go through tiers: the scn client talks to a per-node scion-daemon, which talks to regional scion-cache servers, which talk to scion-server instances backed by Spanner or cloud storage. Anthropic reports over 99% cache hits at each tier. - Branches are mostly gone. Scion "discourages refs (pointers to commits, such as branches and tags)," except for the main ref and release branches, because shared global state is what stops scaling. The results Anthropic reports: an infrastructure footprint "200x smaller than our previous git mirroring infrastructure," checkouts for CI and research jobs that take "seconds rather than minutes," and "a 25% CPU capacity win" for CI alone. These are Anthropic's own numbers on its own systems, and Anthropic hasn't released anything that would let outsiders reproduce them. None of this is new as a category. Anthropic itself points to Google's Piper and Meta's Sapling as earlier server-centric systems. What is new is the reason: Anthropic built one because its own coding model outran its tooling. The process changes matter more than the VCS The part most teams can learn from has nothing to do with storage. Start with the merge queue. "A proper merge queue, for example, doesn't make much sense at agent scale," Anthropic writes. It moved the safety net later instead: "we've invested in post-merge and pre-deploy automated testing." Review changed too. "Our team members still review every change request for the system," but "they now examine the intent of the change as opposed to reading every line of code." Anthropic does not claim the model got everything right. "Agents will sometimes violate core system constraints to satisfy a pattern." One adversarial review "was overly defensive about trusting another internal service," and Claude piled on extra validation to satisfy it. The upside was large too. The team handed an early internal snapshot of Claude Mythos Preview the whole ingestion pipeline to rewrite, and the new version, per Anthropic, "consumed 20x less memory and was 4x faster." Every engineer who worked on the system would now rate Claude as better at writing code than themselves, the post adds. !Abstract illustration of a request travelling down a tiered stack of cache layers, from a small client node through a middle cache ring to a large central server cylinder, with most requests stopping early at the upper tiers Scion's fetch path: client, node daemon, regional cache, server. Most requests never reach the bottom. Should you leave git? Anthropic says no Anthropic is blunt about this. "Scion is by no means a suitable replacement for git for general use." The concepts "really only make sense for truly enormous monorepos." Anthropic has not announced a public release, and it describes its internal systems as highly bespoke. What it asks for instead is better git infrastructure: "Git platforms can provide comprehensive server-side RPCs like Scion," and "A git server doesn't have to work the same way as a git client does." Two of Scion's ideas already exist in stock git at a smaller scale. A partial clone skips file contents until you need them, and sparse checkout only materialises the directories you name. Both are flags on git clone. For a CI job that touches one service: git clone --filter=blob:none --sparse \ https://github.com/your-org/your-repo.git cd your-repo git sparse-checkout set services/billing It won't get you anywhere near Anthropic's numbers, but the instinct is the same: stop shipping the whole repository to a job that reads one folder. My reading of the post: most teams will not hit 30,000 merges a day, but plenty will hit the moment when agents open changes faster than CI and review can absorb them. That is the moment to look at your merge queue and your review rules, long before you look at your VCS. Several agents on one repo, at laptop scale I build Hermes IDE, so weigh this section with that in mind. Hermes is not affiliated with Anthropic. At Anthropic's volume, branches are shared global state worth avoiding. On one machine with three agents, the problem is smaller and different: two agents in one checkout overwrite each other. In Hermes, ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own git worktree, with the task as the agent's first prompt. Claude Code, Codex, Antigravity CLI and other command-line agents each run in a real terminal. The review side is closer to what Anthropic describes. The Review Desk (⌘G) shows an agent's changes by turn or by file, with risk flags, and sends a line comment back to the agent that wrote it. Reviewing one turn at a time makes it easier to judge the intent behind a change. When you are done, Land commits, opens a pull request or merges locally, and can be undone. How many agent-written changes a day can your CI take before it becomes the queue? Tell me in the comments. --- If this was useful Hermes IDE is free, source-available (BSL 1.1), and runs on macOS, Windows and Linux. Download it and give two agents the same ticket on separate branches, or star it on GitHub if keeping agents apart in one repo matters to you. If you build agents yourself, the AI Agents Pocket Guide covers the patterns behind agents that write and review code, like the ones in Anthropic's post. 
From Github to Container: Running a Node.js App (Wonderkid app) Locally and Packaging It with Docker. Getting a project running on your machine and then packaging it in a container are two of the most common tasks in modern development and DevOps work. This guide takes a Node.js web application (wonderkid) from a fresh git clone to a running Docker container, with an explanation of why each step matters. Prerequisites: Git, Node.js (v22 recommended) with npm, VS Code, and Docker Desktop. #Part 1: Local Application Setup Before containerizing anything, confirm the application works on your own machine. If it doesn't run locally, it won't run in a container either, and debugging is much easier outside Docker. Lets follow the following steps to confirm this: ##Step 1: Clone the repository Open VS Code and create a dedicated workspace folder for your projects (in this case, I used the date I created the folder to name the folder ‘oct326’ ). Keeping projects in one place makes them easier to find and manage. Open the integrated terminal (Terminal → New Terminal, or Ctrl + `), navigate to that folder, and clone the repository. To do this you will need the repository url which you will get from the GitHub repository. : git clone <repository-url> git clone https://github.com/your-org/wonderkids.git !git clone This downloads the full project, including its history, into a new folder named after the repository. ##Step 2: Navigate into the project Before navigating into the folder, type the Ls command to see that the folder exists !ls Then use the ‘cd’ command to navigate inside the project folder. cd <folder-name> !Cd Every command from here on must run from the project root, the folder containing package.json. Running ls (macOS/Linux) or dir (Windows) should show package.json, src/ or app/, and similar files. If you don't see package.json, you're in the wrong folder. ##Step 3: Install dependencies npm install !npm install npm reads package.json (and package-lock.json, if present) and downloads everything the project needs into a node_modules/ folder. This can take a minute or two. Some warnings about deprecated packages are normal, but red ERR! lines mean something failed, often a Node version mismatch, so check the project's README for the required version. ##Step 4: Run the app locally npm run dev !npm rundev This starts the development server, which usually prints something like:Local: localhost Open that address in your browser to confirm the application loads. Dev servers typically include hot reloading, so changes to your code appear instantly. !Open When you're finished, stop the server by pressing Ctrl + C in the terminal. Tip: If the port is already in use, another process is occupying it. Stop that process or run the app on a different port, depending on your framework's configuration. #Part 2: Docker Application Containerization Containerization packages your application together with its runtime and dependencies, so it behaves the same on your laptop, a teammate's machine, a CI pipeline, or a cloud host. These are the steps I took to contanerize wonderkid application ##Step 1: Check that Docker Desktop is running Docker commands need the Docker engine running in the background. Launch Docker Desktop and wait until its status shows as running. Then verify from the terminal: docker --version docker info If docker info returns an error like "Cannot connect to the Docker daemon," Docker Desktop isn't running yet. ##Step 2: Create the configuration files From the project root, create the two files Docker needs: touch Dockerfile !touch touch .dockerignore !Docker ignore Alternatively, you can create it manually without using the codes. On Windows PowerShell, use New-Item Dockerfile and New-Item .dockerignore instead, or create the files from VS Code's Explorer panel. Dockerfile is the recipe that tells Docker how to build your image. .dockerignore lists files that should not be sent to Docker during the build. ##Step 3: Configure the .dockerignore !Docker ignore config Without a .dockerignore, Docker copies everything in your folder into the build context, including a large node_modules directory and your Git history. That slows builds and can leak sensitive files. Add: node_modules npm-debug.log .next dist build .git .gitignore .env .env.* Dockerfile .dockerignore README.md .vscode Excluding node_modules is especially important. The container installs its own dependencies for its own operating system, and copying yours in can cause conflicts. ##Step 4: Write the Dockerfile with a multi-stage build !Dockerfile config A multi-stage build uses several FROM stages in one Dockerfile. Build tools and development dependencies are needed to compile your app but not to run it. By building in one stage and copying only the results into a clean final stage, you get a smaller, faster, and more secure image. Here is a solid starting template using Node 22 on Alpine Linux, a minimal base image: Stage 1: Install dependencies ---------- FROM node:22-alpine AS deps WORKDIR /app Copy only dependency manifests first to leverage Docker layer caching COPY package.json package-lock.json ./ RUN npm ci Stage 2: Build the application ---------- FROM node:22-alpine AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . RUN npm run build Stage 3: Production runtime ---------- FROM node:22-alpine AS runner WORKDIR /app ENV NODE_ENV=production ENV PORT=3000 Run as a non-root user for better security RUN addgroup -S appgroup && adduser -S appuser -G appgroup Copy only what is needed to run the app COPY --from=builder /app/package.json ./package.json COPY --from=builder /app/package-lock.json ./package-lock.json COPY --from=builder /app/node_modules ./node_modules COPY --from=builder /app/dist ./dist Remove dev-only packages from the final image RUN npm prune --omit=dev && chown -R appuser:appgroup /app USER appuser EXPOSE 3000 CMD ["npm", "start"] ###Why it's built this way: Choice Reason node:22-alpine A current, actively supported Node version on a small base image with a reduced attack surface. Prefer a maintained LTS release. Three stages Separates dependency install, build, and runtime so the final image carries only what it needs. Copying package*.json first Docker caches layers. If your dependencies haven't changed, rebuilds skip the slow install step. npm ci Installs exact versions from the lockfile, giving reproducible builds (faster and stricter than npm install). Non-root user If the app is ever compromised, the attacker doesn't get root inside the container. EXPOSE 3000 Documents which port the app listens on. Use 3001 if your app runs there. (just as we did in this case) Adapt it to your project. The template above assumes your build outputs to dist/ and npm start runs the production server. Check your package.json scripts and adjust: Next.js: the build output is .next/, so copy .next, public, and package.json instead of dist. Next.js's output: "standalone" option can shrink the image further. Vite/React single-page apps: these produce static files in dist/. A common pattern is a final stage using nginx:alpine that serves the static files, in which case you expose port 80 instead. Express/NestJS APIs: copy your compiled output (often dist/) and set CMD ["node", "dist/main.js"] or similar. Also make sure your app listens on 0.0.0.0 (not just localhost) and respects the PORT environment variable. Otherwise it won't be reachable from outside the container. ##Step 5: Build the Docker image From the project root, run: docker build -t <image-name>:<tag> . For example I used: docker build -t wonderkids:latest . !docker build -t assigns a name and tag to the image. The trailing . tells Docker to use the current directory as the build context. Docker works through each stage and prints progress. When it finishes, confirm the image exists: docker images You'll see your image listed with its size. Compare it to a single-stage build and the multi-stage savings will be clear. Tip: Use meaningful tags such as wonderkids:1.0.0 rather than relying on latest alone, so you can roll back to a known version. ##Step 6: Run the container Start a container from your image, mapping a port on your machine to the port inside the container: docker run -p <host-port>:<container-port> --name <container-name> <image-name>:<tag> For example: docker run -p 3001:3001 --name wonderkids-app wonderkids:latest !Docker run -p 3001:3001 maps host port 3001 to container port 3001. The format is always host:container. To access the app on a different host port, change only the left side: -p 8080:3000 makes it available at localhost. --name gives the container a friendly name instead of a random one. Open localhost in your browser to confirm the containerized app is running. Press Ctrl + C to stop it if it's running in the foreground. Useful variations: Run in the background (detached mode) docker run -d -p 3000:3000 --name wonderkids-app wonderkids:latest Automatically remove the container when it stops docker run --rm -p 3000:3000 wonderkids:latest The container has been created on Dockerhub. ###Everyday Docker Commands Useful for Every DevOps Engineer Task Command List running containers docker ps List all containers docker ps -a View logs docker logs wonderkids-app Follow logs live docker logs -f wonderkids-app Stop a container docker stop wonderkids-app Start a stopped container docker start wonderkids-app Remove a container docker rm wonderkids-app Remove an image docker rmi wonderkids:latest Open a shell inside a container docker exec -it wonderkids-app sh ###Troubleshooting "Cannot connect to the Docker daemon." Docker Desktop isn't running. Start it and wait for it to finish initializing. "Port is already allocated." Something else is using that host port, possibly your local dev server from Part 1 or an older container. Stop it, or map to a different host port (-p 3001:3000). "Container name already in use." A previous container with that name exists. Remove it with docker rm <container-name> or choose a new name. Container starts then exits immediately. Check docker logs <container-name>. Common causes are a wrong CMD, a missing build output folder, or missing environment variables. App runs in the container but the browser can't reach it. Confirm the app binds to 0.0.0.0 rather than 127.0.0.1, and that your -p mapping matches the port the app actually listens on. Build fails at COPY --from=builder /app/dist. Your framework outputs somewhere other than dist/. Check your build script and update the path. ##Conclusion We've now taken an application through a complete workflow: cloned it, installed dependencies, verified it locally, and then packaged it as an optimized, secure Docker image that runs identically anywhere Docker does. From here, natural next steps are pushing the image to a registry (Docker Hub or Azure Container Registry), using Docker Compose for multi-service setups, and automating image builds in a CI/CD pipeline. We will complete this in the next article.
Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計 TL;DR: 結論だけ知りたい方へ（今すぐエラーを防ぐ最小コード） セルの再実行で fatal: destination path 'my-tool' already exists and is not an empty directory. が出て困っている場合は、クローン直前に カレントディレクトリを /content に戻して既存フォルダを削除する 以下の数行をセルの先頭に追加してください。 import os, shutil # 1. カレントディレクトリを必ず /content に戻す（迷子防止） os.chdir("/content") # 2. 既にフォルダが存在していれば削除する（再実行時の衝突防止） TARGET_DIR = "/content/my-tool" if os.path.abspath(TARGET_DIR).startswith("/content/drive"): raise ValueError("❌ TARGET_DIR に Google Drive 配下は指定できません（データ保護）。") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) # 3. クローン実行 !git clone https://github.com/my-org/my-tool.git {TARGET_DIR} > これだけでは不十分なケースとは？ > 個人利用の単発スクリプトであれば上記で足りますが、「モジュール更新がメモリに残って反映されない（sys.modules キャッシュ問題）」「配布相手の環境でタグ不在や通信切断で止まる」「プライベートリポジトリの認証トークンが残る」 といった実践的な課題を解決するには、追加の考慮が必要です。本稿ではこれらを解決する設計とコードを解説します。 :::note info 【Colab実運用基盤シリーズ】 - 第1弾（データ永続化編）: Google ColabでGoogle Drive上のSQLite / DuckDBが遅い・ロック破損する問題と、ローカルディスクを使ったデータステージング設計 - 第2弾（コード配布編・本作）: Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計 - 関連（前史）: Google Colab × SQLite3 を「壊れる前提」で安定運用する方法（自己修復20行＋AIキャッシュ戦略） ::: --- はじめに Google Colaboratory（以下、Colab）は、ブラウザだけで Python コードを実行できるため、オープンソースツールや分析パイプライン（第1弾で触れた株価スクリーナーなど）をチームメンバーや一般ユーザーに配布する手段として広く使われています。 その際、最も一般的な配布方法は「ノートブック内に !git clone https://github.com/... を記述し、リポジトリのコードをそのまま実行させる」という構成です。しかし、この単純な git clone をそのまま配ると、セルの再実行時やリリースのタイミングでエラーが発生することがあり、パイプラインが停止します。 本稿では、Colab 上で GitHub リポジトリのコードと依存ライブラリを確実に立ち上げるための環境セットアップ処理をまとめます。 --- 第1章：よくある書き方と、見落としていたこと GitHub で管理されているコードを Colab 上で利用者に実行させる際、多くのチュートリアルでは以下のようなシンプルなセルが提示されます。 # 初期の典型的なセットアップセル !git clone -b v1.0.0 https://github.com/my-org/my-tool.git %cd my-tool !pip install -r requirements.txt from src.main import run_pipeline run_pipeline() この書き方には、次のような都合のよい想定（思い込み）がありました。 - 「セルを上から順に1度だけ実行する」という想定: セルの再実行や、途中でエラーになってやり直す操作を考えていない - 「指定したタグは必ず存在する」という想定: リモートにそのタグが確実にプッシュされていると思い込んでいる（push 漏れやタイポを考慮していない） - 「環境は最初から綺麗である」という想定: %cd によるカレントディレクトリの移動やインストール状態が、再実行時にも都合よく保たれると思い込んでいる --- 第2章：本番運用で起きた3つのトラブル 配布を想定した動作検証や、テスト配布を行った際、以下のトラブルに遭遇しました。 1. セル再実行時のディレクトリ衝突エラー エラー発生やパラメータ変更で利用者がセットアップセルをもう一度実行した際、Git が衝突エラーを起こして停止しました。 fatal: destination path 'my-tool' already exists and is not an empty directory. 利用者は手動でファイルを消す方法がわからず、ランタイムの全初期化を強いられます。 2. リリース直後のタグ不一致によるエラー中断 新バージョン（例: v1.3.1）のリリース告知直後に、利用者がクローンを実行すると以下のエラーで停止しました。 fatal: Remote branch v1.3.1 not found in upstream origin この原因の多くは、開発者側で git push 時にタグの送信を忘れていた（git push --tags の実行漏れ）か、ノートブック側のパラメータに入力したタグ名のタイポでした。このような人為的ミスや指定の不整合により、パイプライン全体が中断し、手動対応が必要になりました。 3. 作業ディレクトリ消失によるモジュール見失い セル内で %cd my-tool を実行した状態で、前述のエラー復旧や再実行のために外側から shutil.rmtree("my-tool") が走ると、Python プロセスのカレントディレクトリが存在しない無効なパスを指す状態になります。その結果、以降の相対パス操作やファイル読み込みで FileNotFoundError が発生しました。また、sys.modules に過去の古いモジュールがキャッシュされ、再取得したコードがメモリ上に反映されない不整合も起きました。 --- 第3章：なぜ単純な git clone では失敗するのか 起きていた問題の原因を掘り下げると、Colab という対話的実行環境の特性に対する考慮が不足していました。 1. セルの再実行（やり直し）を考慮していない Colab は利用者が試行錯誤しながらセルを何度も再実行する環境です。クローン先にフォルダが既に存在するかどうかを気にせず、常に同じ名前で git clone を実行するコードは、やり直し（再実行）に耐えられません。 2. タグの直書きによるエラー停止 特定のタグ（--branch v1.3.1）をコード内に直書きしていると、開発者のタグ push 忘れや、利用者のタイポが1箇所あるだけでスクリプト全体が停止してしまいます。通信エラーなのか、タグが存在しないのかを判別できず、代替ブランチへの切り替えもできない構成だったため、すぐに処理が止まってしまっていました。 3. カレントディレクトリやモジュールキャッシュがプロセス内に残り続ける問題 Colab ノートブックのカーネルは、単一の持続的な Python プロセス上で対話的に実行されます。シェルコマンドの %cd や import による状態変更はプロセス全体に残り続けるため、以下のトラブルを引き起こします。 1. 滞在中のフォルダを削除してしまう（カレントディレクトリの喪失）: セル内で %cd my-tool を実行した後に、再クローン目的で外側から shutil.rmtree("my-tool") を実行すると、Python プロセス自身がいまいる場所（カレントディレクトリ）がディスク上から消えてしまいます。この状態になると、以降の相対パス指定やファイルの読み書きで FileNotFoundError が発生します。 2. sys.modules によるモジュールキャッシュの残存: Python のインポート機構（importlib）は、一度読み込んだモジュールをプロセス空間の辞書 sys.modules にキャッシュします。ファイルシステム上の .py ファイルを新バージョンでクローンし直しても、Python はディスクを再走査せずメモリ上の古いモジュールを返し続けます。その結果、セルの再実行を行っても最新のバグ修正が反映されず、新設された関数が見つからない AttributeError や古いロジックによる不整合が発生します。 --- 第4章：どう設計し直したか（再実行に強い環境セットアップ） これらの問題を踏まえ、利用者がセルの再実行を行っても安定して起動するセットアップ処理を設計し直しました。 設計の基本方針 - 作業ディレクトリのルート固定（/content に戻す）: 操作の直前に必ずカレントディレクトリを /content に戻す。既存の展開先ディレクトリは事前にクリーンアップし、常にクリーンな状態からクローンを開始する - 最新コミットだけの取得（git clone --depth 1）: 過去のコミット履歴をダウンロードせず、指定した最新のスナップショットのみを取得して通信量と所要時間を抑える - タグの事前確認と main ブランチへの切り替え（代替取得）: git ls-remote でタグの存在を確認し、見つからない場合はフラグ（許可時のみ）に応じて main ブランチに切り替えて取得を継続する - 動的インポートとモジュールキャッシュのリセット: sys.modules 内の自作パッケージを明示的にクリアし、再実行時でも新しくクローンされたコードをメモリに再ロードする flowchart TD Start["Step 1 実行開始"] --> CD["os.chdir('/content')<br/>(カレントを /content に戻す)"] CD --> CheckDir{"TARGET_DIR が<br/>既に存在するか？"} CheckDir -- "Yes" --> Clean["shutil.rmtree()<br/>(既存ディレクトリを事前消去)"] CheckDir -- "No" --> Verify Clean --> Verify["check_tag() でリモート確認"] Verify -- "通信エラー (error)" --> Retry["指数バックオフで再試行 (最大3回)"] Retry --> Verify Verify -- "再試行上限超過" --> ErrorExit["RuntimeError で明示的中断<br/>(GitHub接続障害案内)"] Verify -- "タグ存在 (found)" --> CloneTag["Attempt: git clone --depth 1<br/>--branch TARGET_TAG (指定タグ)"] Verify -- "タグ不在 (not_found)" --> CheckAllow{"ALLOW_MAIN_FALLBACK<br/>が有効か？"} CheckAllow -- "No (無効)" --> TagExit["RuntimeError で中断<br/>(タグ名確認案内)"] CheckAllow -- "Yes (有効)" --> CloneMain["警告出力 ＆ ブランチを 'main' に切替<br/>git clone --branch main"] CloneTag --> Pip["sys.executable -m pip install -q -r requirements.txt"] CloneMain --> Pip Pip --> Reload["sys.modules から自作パッケージを破棄<br/>(最新モジュールの強制再読込)"] Reload --> Ready["環境セットアップ完了 (次Stepへ)"] style Start fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333 style Pip fill:#f0fff0,stroke:#2a2,stroke-width:1px,color:#333 style Ready fill:#d0ffd0,stroke:#2a2,stroke-width:2px,color:#333 style ErrorExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333 style TagExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333 --- 第5章：実装のポイントとコード ここまでに整理した設計を、Google Colab 上でそのまま動作確認できるようにまとめた動作確認用のセットアップコードです。 ノートブックの1つのセルに貼り付け、上部のパラメータ（REPO_URL やタグ名）をご自身のリポジトリに合わせて設定して実行してください。クローン、依存関係のインストール、モジュールキャッシュの破棄、パス解決までが一気通貫で完了します。 # @title 【Step 1】環境セットアップ # @markdown 配布したいリポジトリの情報を設定してください REPO_URL = "https://github.com/my-org/my-tool.git" # @param {type:"string"} TARGET_TAG = "v1.3.1" # @param {type:"string"} ALLOW_MAIN_FALLBACK = False # @param {type:"boolean"} # ↑ タグ不在時に最新開発版 (main) での実行を許容する場合は True にチェック TARGET_DIR = "/content/my-tool" PACKAGE_NAME = "src" # 読み込むパッケージフォルダ名（例: src または my_tool） import importlib import os import re import shutil import subprocess import sys import time # 1. 作業ディレクトリを必ず /content に戻す os.chdir("/content") # 2. 既存ディレクトリが存在する場合は削除 (再実行時の衝突防止) # ※Drive配下の誤削除を防ぐためガードを配置 if os.path.abspath(TARGET_DIR).startswith("/content/drive"): raise ValueError("❌ TARGET_DIR に Google Drive 配下のパスは指定できません（データ保護のため）。") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) # 3. リモートタグの事前確認（3状態判定と指数バックオフ） def check_tag(repo_url: str, tag: str, retries: int = 3) -> str: """'found' / 'not_found' / 'error' を返す。通信エラーは指数バックオフで再試行""" git_env = os.environ.copy() git_env["GIT_TERMINAL_PROMPT"] = "0" # 認証プロンプトによる無応答停止を防止 for i in range(retries): try: res = subprocess.run( ["git", "ls-remote", "--exit-code", repo_url, f"refs/tags/{tag}"], capture_output=True, text=True, timeout=10, env=git_env, ) if res.returncode == 0: return "found" if res.returncode == 2: # 通信成功・該当refなし return "not_found" except (subprocess.TimeoutExpired, subprocess.CalledProcessError): pass time.sleep(2 ** i) return "error" status = check_tag(REPO_URL, TARGET_TAG) if status == "error": raise RuntimeError( "❌ リポジトリが見つからないか、GitHub に接続できません。\n" "上部の 'REPO_URL' を実際のリポジトリURLに書き換えて再実行してください。" ) if status == "not_found": if not ALLOW_MAIN_FALLBACK: raise RuntimeError(f"❌ タグ '{TARGET_TAG}' が見つかりません。タグ名を確認してください。") print(f"⚠️ タグ '{TARGET_TAG}' がないため main で続行します（未検証コードの可能性あり）。") target_to_clone = "main" else: target_to_clone = TARGET_TAG # 4. 最新コミットだけの取得 (--depth 1) clone_success = False git_env = os.environ.copy() git_env["GIT_TERMINAL_PROMPT"] = "0" for attempt in range(1, 3): try: print(f"📥 リポジトリを取得中 (attempt {attempt}/2, target: {target_to_clone})...") subprocess.run( ["git", "clone", "--depth", "1", "--branch", target_to_clone, REPO_URL, TARGET_DIR], check=True, capture_output=True, text=True, env=git_env, ) clone_success = True break except subprocess.CalledProcessError as e: # トークン等を含むコマンド全文ではなく、標準エラーのみを安全に出力（URLトークンは伏字化） raw_msg = e.stderr.strip() if e.stderr else "クローン処理に失敗しました" err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg) print(f"⚠️ Git クローン警告 (attempt {attempt}): {err_msg}") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) time.sleep(2) if not clone_success: raise RuntimeError("❌ リポジトリ取得に失敗しました。ネットワーク接続または GitHub の稼働状況を確認してください。") # 5. 依存ライブラリの導入 (カレントディレクトリを移動せず、sys.executable と絶対パスで指定) req_path = os.path.join(TARGET_DIR, "requirements.txt") if os.path.exists(req_path): try: print("📦 依存ライブラリをインストール中...") subprocess.run( [sys.executable, "-m", "pip", "install", "-q", "-r", req_path], check=True, capture_output=True, text=True, ) except subprocess.CalledProcessError as e: err_msg = e.stderr.strip() if e.stderr else "pip インストールに失敗しました" raise RuntimeError(f"❌ 依存ライブラリのインストールに失敗しました: {err_msg}") from None # 6. 自作モジュールのキャッシュを破棄 (再実行時の古いコード混入を防止) for mod in list(sys.modules.keys()): if mod == PACKAGE_NAME or mod.startswith(f"{PACKAGE_NAME}."): del sys.modules[mod] importlib.invalidate_caches() # ファインダーの内部ファイル検索キャッシュをリセット # パスを追加してインポートを有効化 if TARGET_DIR not in sys.path: sys.path.insert(0, TARGET_DIR) # 7. エントリポイントのインポート確認 try: main_mod = importlib.import_module(f"{PACKAGE_NAME}.main") run_pipeline = getattr(main_mod, "run_pipeline", None) print(f"✅ エントリポイント（{PACKAGE_NAME}.main）のインポートに成功しました。") except ModuleNotFoundError as e: # パッケージまたは main モジュール自体が存在しない場合のみスキップ if e.name in (PACKAGE_NAME, f"{PACKAGE_NAME}.main"): print(f"ℹ️ {PACKAGE_NAME}.main が見つからないため、インポートをスキップしました（ご自身のリポジトリ構造に合わせて書き換えてください）。") else: # 依存ライブラリ等の読み込み失敗は本物のエラーとして再送出 raise print("✅ セットアップが完了しました。") 💡 このコードを配布用ノートブックに組み込む手順 1. 新しいノートブックを作成: 最初のセルに上記コードをそのまま貼り付けます。 2. パラメータを3箇所変更: - REPO_URL: ご自身の GitHub リポジトリ URL（例: https://github.com/your-org/your-repo.git） - TARGET_TAG: 配布したい安定版のタグ名（例: v1.0.0） - PACKAGE_NAME: リポジトリ内の自作モジュール名（例: src や my_tool） 3. フォームを整えてコードを隠す（任意）: - セルの右上メニュー（縦三点リーダー）から「フォーム」➔「コードを非表示」を選択すると、一般利用者の画面には生コードが隠れ、タイトルと入力フォーム（実行ボタン）だけが表示された綺麗な UI として配布できます。 4. 後続セル（Step 2）から実行: - 次のセル（Step 2）を作成し、from {PACKAGE_NAME}.main import run_pipeline; run_pipeline()（例: from src.main import run_pipeline; run_pipeline()）のように呼び出せば、クローンされたコードがそのまま動作します。 :::note info パッケージ名（ここでは src）は、サードパーティ製ライブラリと重複しない固有のパッケージ名（例: my_tool）にしておくと、名前空間の衝突をより確実に防ぐことができます。 ::: 2. コラム：プライベートリポジトリを安全に扱うには？ もしコードを一般公開せず、チームメンバーや特定利用者向けに配布する場合、トークンをハードコードしてはいけません。Colab 標準の 「Secrets（🔑）」機能 を使って環境変数経由でトークンを注入し、クローン完了後にローカルの Git 設定からトークンを抹消します。 # プライベートリポジトリの場合の安全なトークン注入と痕跡消去 import re import subprocess from google.colab import userdata try: gh_token = userdata.get("GITHUB_TOKEN") auth_repo_url = f"https://x-access-token:{gh_token}@github.com/my-org/private-tool.git" except Exception: raise RuntimeError("🔑 左メニューの鍵アイコン (Secrets) から 'GITHUB_TOKEN' を設定してください。") # クローン実行（エラー出力時にトークンが露出しないよう URL 部分を伏字化） try: subprocess.run( ["git", "clone", "--depth", "1", "--branch", TARGET_TAG, auth_repo_url, TARGET_DIR], check=True, capture_output=True, text=True, ) # クローン成功後、ローカル .git/config 内の URL からトークンを即時除去 clean_repo_url = "https://github.com/my-org/private-tool.git" subprocess.run( ["git", "-C", TARGET_DIR, "remote", "set-url", "origin", clean_repo_url], check=True, ) except subprocess.CalledProcessError as e: raw_msg = e.stderr.strip() if e.stderr else "認証またはリポジトリの取得に失敗しました" err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg) raise RuntimeError(f"❌ プライベートリポジトリの取得エラー: {err_msg}") from None --- 第6章：動かしてみた結果とベンチマーク このセットアップ処理の導入により、運用の安定性が向上しました。 1. 起動時間とエラー耐性の比較 数百コミット規模のリポジトリを Colab 無料枠でクローンした筆者環境での実測値です。 | 項目 / 状態 | 通常の !git clone | 改善したセットアップ | 改善効果 | | :--------------------------- | :---------------------------: | :----------------------------------: | :------------------------- | | 初動の転送時間 | 約 5〜10 秒（全履歴取得時） | 約 2〜4 秒（--depth 1） | 初動の短縮 | | 同一セルの再実行 | already exists でエラー停止 | 自動消去・再取得で衝突なし | 冪等性の担保 | | タグ push 漏れ・タイポ時 | エラーによる処理中断 | 検証を経て許可時のみ main 退避 | タグ不在時に main 代替取得 | | 自作モジュール更新 | 再起動しないと旧コード参照 | sys.modules 破棄で即反映 | コード不整合の防止 | 2. 同一ランタイムでのセル再実行に対する冪等性と復旧 セッションが完全に切断された場合は Colab の VM ごと初期化されるため前回の残骸は残りませんが、実際の運用で頻繁に発生するのは「エラー発生後やパラメータ変更時に、同一ランタイム上でセルをもう一度実行する」ケースです。 セットアップコードの冒頭でカレントディレクトリを /content に戻し、既存フォルダを消去することで、同じセッション内でのやり直し時にディレクトリの衝突や、削除済みフォルダ内に留まって迷子になる問題を防ぎます。 ※なお、pip install の実行途中で処理を強制中断した場合、site-packages 配下に不完全なパッケージが残存する可能性があります。その場合はノートブックのメニューからランタイムの再起動（Restart session）を行うか、pip install --force-reinstall を実行して修復します。 3. 今回の設計が持つトレードオフと運用上の制約 本設計は「一般利用者の手元で止まりにくく、安全に再実行できること」を優先しています。そのため、以下のトレードオフ（何を得て、何を妥協したか）が存在します。 - 【起動速度 vs Git履歴の閲覧性（--depth 1 の代償）】最新コミットに絞ることで履歴追跡は切り捨てる: --depth 1 によって数秒での高速クローンを実現した代償として、過去のコミット履歴（git log や git diff、過去リビジョンへのチェックアウト）は利用できません。コードを配布して実行させる用途に特化し、開発者向けの履歴閲覧性は意図して捨てています - 【可用性（止まらないこと） vs コードの再現性】main への代替切り替えは未検証コードのリスクを伴う: タグ不在時に main ブランチへ自動退避して処理を継続させる選択肢を用意した代償として、開発途中の未検証コードが動いてしまうリスクを負います。そのため本設計では ALLOW_MAIN_FALLBACK をデフォルトで False とし、利用者がリスクを承知の上で明示的に有効化した場合のみ main へ切り替える安全弁を設けています - 【高速な再読込 vs C拡張モジュールの更新限界】インメモリ破棄で扱えるのは純粋な Python コードに限定される: ランタイム再起動を挟まずに sys.modules の破棄で最新コードを即時反映させる軽快さを取った代償として、numpy や torch など一度メモリに共有ライブラリ（.so）がロードされたバイナリ拡張パッケージの更新は反映できません。これらをアップグレードした場合は、OS プロセス自体の再起動（ランタイム再起動）が必要です - 【安全なトークン注入 vs ログ出力・プロセス一覧からの漏洩リスク】例外メッセージのサニタイズが必要になる: Secrets を用いてプライベートリポジトリを安全にクローンできる利便性を取った代償として、例外発生時にトークンが露出しないよう stderr のサニタイズ（伏字化）が必要です。また、クローン実行中は OS のプロセス一覧（ps）から一時的に引数のトークンが見える点にも留意が必要です --- 結び Google Colab 上でリポジトリのコードを動かす作業は、ローカル開発環境での git clone とは前提が異なります。 「作業ディレクトリを /content に戻す」「既存フォルダの事前クリーンアップ」「--depth 1 による最新取得」「タグ不在時の main 代替取得」「モジュールキャッシュの破棄」という一連の対策をセットアップセルに組み込むことで、ボタン1つで安定して起動できる配布基盤を構築できます。 本稿で確立したコード展開フローと、第1弾（データ永続化編）で設計した 2 層ストレージフローを組み合わせることで、ノートブック全体のライフサイクルは以下のように整理されます。 > 【Step 1】環境セットアップ（本稿：コード展開・依存解決） > ↓ > 【Step 2】Pull Phase（第1弾：永続層 Drive から作業層ローカルディスクへデータステージング展開） > ↓ > 【Step 3】処理実行（ローカルディスク上での高速バッチ・クエリ処理） > ↓ > 【Step 4】Push Phase（第1弾：成果物先行・DB末尾置換でデータステージング書き戻し ＆ アンマウント） 2 つの設計を組み合わせることで、対話的なノートブックであっても運用に耐えうる安定したバッチ実行基盤として活用できるようになります。 この記事が参考になったら、ストックしておくとシリーズの続きを追いやすくなります。 質問や、別の環境で試した結果はコメントで教えてください。 --- > 免責事項（Disclaimer） > 本記事に掲載されているコードや設定例は筆者の検証環境に基づくものであり、外部サービス（Google Colab、GitHub 等）の仕様変更や実行環境の違い等によって生じたいかなる損害についても責任を負いかねます。実際の運用や設定はご自身の責任において確認の上で行ってください。
Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計 TL;DR: 結論だけ知りたい方へ（今すぐエラーを防ぐ最小コード） セルの再実行で fatal: destination path 'my-tool' already exists and is not an empty directory. が出て困っている場合は、クローン直前に カレントディレクトリを /content に戻して既存フォルダを削除する 以下の数行をセルの先頭に追加してください。 import os, shutil # 1. カレントディレクトリを必ず /content に戻す（迷子防止） os.chdir("/content") # 2. 既にフォルダが存在していれば削除する（再実行時の衝突防止） TARGET_DIR = "/content/my-tool" if os.path.abspath(TARGET_DIR).startswith("/content/drive"): raise ValueError("❌ TARGET_DIR に Google Drive 配下は指定できません（データ保護）。") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) # 3. クローン実行 !git clone https://github.com/my-org/my-tool.git {TARGET_DIR} > これだけでは不十分なケースとは？ > 個人利用の単発スクリプトであれば上記で足りますが、「モジュール更新がメモリに残って反映されない（sys.modules キャッシュ問題）」「配布相手の環境でタグ不在や通信切断で止まる」「プライベートリポジトリの認証トークンが残る」 といった実践的な課題を解決するには、追加の考慮が必要です。本稿ではこれらを解決する設計とコードを解説します。 :::note info 【Colab実運用基盤シリーズ】 - 第1弾（データ永続化編）: Google ColabでGoogle Drive上のSQLite / DuckDBが遅い・ロック破損する問題と、ローカルディスクを使ったデータステージング設計 - 第2弾（コード配布編・本作）: Colabで再実行時の「fatal: destination path already exists」を防ぐ：GitHubリポジトリの安全なクローン・実行セットアップ設計 ::: --- はじめに Google Colaboratory（以下、Colab）は、ブラウザだけで Python コードを実行できるため、オープンソースツールや分析パイプライン（第1弾で触れた株価スクリーナーなど）をチームメンバーや一般ユーザーに配布する手段として広く使われています。 その際、最も一般的な配布方法は「ノートブック内に !git clone https://github.com/... を記述し、リポジトリのコードをそのまま実行させる」という構成です。しかし、この単純な git clone をそのまま配ると、セルの再実行時やリリースのタイミングでエラーが発生することがあり、パイプラインが停止します。 本稿では、Colab 上で GitHub リポジトリのコードと依存ライブラリを確実に立ち上げるための環境セットアップ処理をまとめます。 --- 第1章：よくある書き方と、見落としていたこと GitHub で管理されているコードを Colab 上で利用者に実行させる際、多くのチュートリアルでは以下のようなシンプルなセルが提示されます。 # 初期の典型的なセットアップセル !git clone -b v1.0.0 https://github.com/my-org/my-tool.git %cd my-tool !pip install -r requirements.txt from src.main import run_pipeline run_pipeline() この書き方には、次のような都合のよい想定（思い込み）がありました。 - 「セルを上から順に1度だけ実行する」という想定: セルの再実行や、途中でエラーになってやり直す操作を考えていない - 「指定したタグは必ず存在する」という想定: リモートにそのタグが確実にプッシュされていると思い込んでいる（push 漏れやタイポを考慮していない） - 「環境は最初から綺麗である」という想定: %cd によるカレントディレクトリの移動やインストール状態が、再実行時にも都合よく保たれると思い込んでいる --- 第2章：本番運用で起きた3つのトラブル 配布を想定した動作検証や、テスト配布を行った際、以下のトラブルに遭遇しました。 1. セル再実行時のディレクトリ衝突エラー エラー発生やパラメータ変更で利用者がセットアップセルをもう一度実行した際、Git が衝突エラーを起こして停止しました。 fatal: destination path 'my-tool' already exists and is not an empty directory. 利用者は手動でファイルを消す方法がわからず、ランタイムの全初期化を強いられます。 2. リリース直後のタグ不一致によるエラー中断 新バージョン（例: v1.3.1）のリリース告知直後に、利用者がクローンを実行すると以下のエラーで停止しました。 fatal: Remote branch v1.3.1 not found in upstream origin この原因の多くは、開発者側で git push 時にタグの送信を忘れていた（git push --tags の実行漏れ）か、ノートブック側のパラメータに入力したタグ名のタイポでした。このような人為的ミスや指定の不整合により、パイプライン全体が中断し、手動対応が必要になりました。 3. 作業ディレクトリ消失によるモジュール見失い セル内で %cd my-tool を実行した状態で、前述のエラー復旧や再実行のために外側から shutil.rmtree("my-tool") が走ると、Python プロセスのカレントディレクトリが存在しない無効なパスを指す状態になります。その結果、以降の相対パス操作やファイル読み込みで FileNotFoundError が発生しました。また、sys.modules に過去の古いモジュールがキャッシュされ、再取得したコードがメモリ上に反映されない不整合も起きました。 --- 第3章：なぜ単純な git clone では失敗するのか 起きていた問題の原因を掘り下げると、Colab という対話的実行環境の特性に対する考慮が不足していました。 1. セルの再実行（やり直し）を考慮していない Colab は利用者が試行錯誤しながらセルを何度も再実行する環境です。クローン先にフォルダが既に存在するかどうかを気にせず、常に同じ名前で git clone を実行するコードは、やり直し（再実行）に耐えられません。 2. タグの直書きによるエラー停止 特定のタグ（--branch v1.3.1）をコード内に直書きしていると、開発者のタグ push 忘れや、利用者のタイポが1箇所あるだけでスクリプト全体が停止してしまいます。通信エラーなのか、タグが存在しないのかを判別できず、代替ブランチへの切り替えもできない構成だったため、すぐに処理が止まってしまっていました。 3. カレントディレクトリやモジュールキャッシュがプロセス内に残り続ける問題 Colab ノートブックのカーネルは、単一の持続的な Python プロセス上で対話的に実行されます。シェルコマンドの %cd や import による状態変更はプロセス全体に残り続けるため、以下のトラブルを引き起こします。 1. 滞在中のフォルダを削除してしまう（カレントディレクトリの喪失）: セル内で %cd my-tool を実行した後に、再クローン目的で外側から shutil.rmtree("my-tool") を実行すると、Python プロセス自身がいまいる場所（カレントディレクトリ）がディスク上から消えてしまいます。この状態になると、以降の相対パス指定やファイルの読み書きで FileNotFoundError が発生します。 2. sys.modules によるモジュールキャッシュの残存: Python のインポート機構（importlib）は、一度読み込んだモジュールをプロセス空間の辞書 sys.modules にキャッシュします。ファイルシステム上の .py ファイルを新バージョンでクローンし直しても、Python はディスクを再走査せずメモリ上の古いモジュールを返し続けます。その結果、セルの再実行を行っても最新のバグ修正が反映されず、新設された関数が見つからない AttributeError や古いロジックによる不整合が発生します。 --- 第4章：どう設計し直したか（再実行に強い環境セットアップ） これらの問題を踏まえ、利用者がセルの再実行を行っても安定して起動するセットアップ処理を設計し直しました。 設計の基本方針 - 作業ディレクトリのルート固定（/content に戻す）: 操作の直前に必ずカレントディレクトリを /content に戻す。既存の展開先ディレクトリは事前にクリーンアップし、常にクリーンな状態からクローンを開始する - 最新コミットだけの取得（git clone --depth 1）: 過去のコミット履歴をダウンロードせず、指定した最新のスナップショットのみを取得して通信量と所要時間を抑える - タグの事前確認と main ブランチへの切り替え（代替取得）: git ls-remote でタグの存在を確認し、見つからない場合はフラグ（許可時のみ）に応じて main ブランチに切り替えて取得を継続する - 動的インポートとモジュールキャッシュのリセット: sys.modules 内の自作パッケージを明示的にクリアし、再実行時でも新しくクローンされたコードをメモリに再ロードする flowchart TD Start["Step 1 実行開始"] --> CD["os.chdir('/content')<br/>(カレントを /content に戻す)"] CD --> CheckDir{"TARGET_DIR が<br/>既に存在するか？"} CheckDir -- "Yes" --> Clean["shutil.rmtree()<br/>(既存ディレクトリを事前消去)"] CheckDir -- "No" --> Verify Clean --> Verify["check_tag() でリモート確認"] Verify -- "通信エラー (error)" --> Retry["指数バックオフで再試行 (最大3回)"] Retry --> Verify Verify -- "再試行上限超過" --> ErrorExit["RuntimeError で明示的中断<br/>(GitHub接続障害案内)"] Verify -- "タグ存在 (found)" --> CloneTag["Attempt: git clone --depth 1<br/>--branch TARGET_TAG (指定タグ)"] Verify -- "タグ不在 (not_found)" --> CheckAllow{"ALLOW_MAIN_FALLBACK<br/>が有効か？"} CheckAllow -- "No (無効)" --> TagExit["RuntimeError で中断<br/>(タグ名確認案内)"] CheckAllow -- "Yes (有効)" --> CloneMain["警告出力 ＆ ブランチを 'main' に切替<br/>git clone --branch main"] CloneTag --> Pip["sys.executable -m pip install -q -r requirements.txt"] CloneMain --> Pip Pip --> Reload["sys.modules から自作パッケージを破棄<br/>(最新モジュールの強制再読込)"] Reload --> Ready["環境セットアップ完了 (次Stepへ)"] style Start fill:#f9f9f9,stroke:#333,stroke-width:1px,color:#333 style Pip fill:#f0fff0,stroke:#2a2,stroke-width:1px,color:#333 style Ready fill:#d0ffd0,stroke:#2a2,stroke-width:2px,color:#333 style ErrorExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333 style TagExit fill:#ffe0e0,stroke:#d33,stroke-width:2px,color:#333 --- 第5章：実装のポイントとコード ここまでに整理した設計を、Google Colab 上でそのまま動作確認できるようにまとめた動作確認用のセットアップコードです。 ノートブックの1つのセルに貼り付け、上部のパラメータ（REPO_URL やタグ名）をご自身のリポジトリに合わせて設定して実行してください。クローン、依存関係のインストール、モジュールキャッシュの破棄、パス解決までが一気通貫で完了します。 # @title 【Step 1】環境セットアップ # @markdown 配布したいリポジトリの情報を設定してください REPO_URL = "https://github.com/my-org/my-tool.git" # @param {type:"string"} TARGET_TAG = "v1.3.1" # @param {type:"string"} ALLOW_MAIN_FALLBACK = False # @param {type:"boolean"} # ↑ タグ不在時に最新開発版 (main) での実行を許容する場合は True にチェック TARGET_DIR = "/content/my-tool" PACKAGE_NAME = "src" # 読み込むパッケージフォルダ名（例: src または my_tool） import importlib import os import re import shutil import subprocess import sys import time # 1. 作業ディレクトリを必ず /content に戻す os.chdir("/content") # 2. 既存ディレクトリが存在する場合は削除 (再実行時の衝突防止) # ※Drive配下の誤削除を防ぐためガードを配置 if os.path.abspath(TARGET_DIR).startswith("/content/drive"): raise ValueError("❌ TARGET_DIR に Google Drive 配下のパスは指定できません（データ保護のため）。") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) # 3. リモートタグの事前確認（3状態判定と指数バックオフ） def check_tag(repo_url: str, tag: str, retries: int = 3) -> str: """'found' / 'not_found' / 'error' を返す。通信エラーは指数バックオフで再試行""" git_env = os.environ.copy() git_env["GIT_TERMINAL_PROMPT"] = "0" # 認証プロンプトによる無応答停止を防止 for i in range(retries): try: res = subprocess.run( ["git", "ls-remote", "--exit-code", repo_url, f"refs/tags/{tag}"], capture_output=True, text=True, timeout=10, env=git_env, ) if res.returncode == 0: return "found" if res.returncode == 2: # 通信成功・該当refなし return "not_found" except (subprocess.TimeoutExpired, subprocess.CalledProcessError): pass time.sleep(2 ** i) return "error" status = check_tag(REPO_URL, TARGET_TAG) if status == "error": raise RuntimeError( "❌ リポジトリが見つからないか、GitHub に接続できません。\n" "上部の 'REPO_URL' を実際のリポジトリURLに書き換えて再実行してください。" ) if status == "not_found": if not ALLOW_MAIN_FALLBACK: raise RuntimeError(f"❌ タグ '{TARGET_TAG}' が見つかりません。タグ名を確認してください。") print(f"⚠️ タグ '{TARGET_TAG}' がないため main で続行します（未検証コードの可能性あり）。") target_to_clone = "main" else: target_to_clone = TARGET_TAG # 4. 最新コミットだけの取得 (--depth 1) clone_success = False git_env = os.environ.copy() git_env["GIT_TERMINAL_PROMPT"] = "0" for attempt in range(1, 3): try: print(f"📥 リポジトリを取得中 (attempt {attempt}/2, target: {target_to_clone})...") subprocess.run( ["git", "clone", "--depth", "1", "--branch", target_to_clone, REPO_URL, TARGET_DIR], check=True, capture_output=True, text=True, env=git_env, ) clone_success = True break except subprocess.CalledProcessError as e: # トークン等を含むコマンド全文ではなく、標準エラーのみを安全に出力（URLトークンは伏字化） raw_msg = e.stderr.strip() if e.stderr else "クローン処理に失敗しました" err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg) print(f"⚠️ Git クローン警告 (attempt {attempt}): {err_msg}") if os.path.exists(TARGET_DIR): shutil.rmtree(TARGET_DIR, ignore_errors=True) time.sleep(2) if not clone_success: raise RuntimeError("❌ リポジトリ取得に失敗しました。ネットワーク接続または GitHub の稼働状況を確認してください。") # 5. 依存ライブラリの導入 (カレントディレクトリを移動せず、sys.executable と絶対パスで指定) req_path = os.path.join(TARGET_DIR, "requirements.txt") if os.path.exists(req_path): try: print("📦 依存ライブラリをインストール中...") subprocess.run( [sys.executable, "-m", "pip", "install", "-q", "-r", req_path], check=True, capture_output=True, text=True, ) except subprocess.CalledProcessError as e: err_msg = e.stderr.strip() if e.stderr else "pip インストールに失敗しました" raise RuntimeError(f"❌ 依存ライブラリのインストールに失敗しました: {err_msg}") from None # 6. 自作モジュールのキャッシュを破棄 (再実行時の古いコード混入を防止) for mod in list(sys.modules.keys()): if mod == PACKAGE_NAME or mod.startswith(f"{PACKAGE_NAME}."): del sys.modules[mod] importlib.invalidate_caches() # ファインダーの内部ファイル検索キャッシュをリセット # パスを追加してインポートを有効化 if TARGET_DIR not in sys.path: sys.path.insert(0, TARGET_DIR) # 7. エントリポイントのインポート確認 try: main_mod = importlib.import_module(f"{PACKAGE_NAME}.main") run_pipeline = getattr(main_mod, "run_pipeline", None) print(f"✅ エントリポイント（{PACKAGE_NAME}.main）のインポートに成功しました。") except ModuleNotFoundError as e: # パッケージまたは main モジュール自体が存在しない場合のみスキップ if e.name in (PACKAGE_NAME, f"{PACKAGE_NAME}.main"): print(f"ℹ️ {PACKAGE_NAME}.main が見つからないため、インポートをスキップしました（ご自身のリポジトリ構造に合わせて書き換えてください）。") else: # 依存ライブラリ等の読み込み失敗は本物のエラーとして再送出 raise print("✅ セットアップが完了しました。") 💡 このコードを配布用ノートブックに組み込む手順 1. 新しいノートブックを作成: 最初のセルに上記コードをそのまま貼り付けます。 2. パラメータを3箇所変更: - REPO_URL: ご自身の GitHub リポジトリ URL（例: https://github.com/your-org/your-repo.git） - TARGET_TAG: 配布したい安定版のタグ名（例: v1.0.0） - PACKAGE_NAME: リポジトリ内の自作モジュール名（例: src や my_tool） 3. フォームを整えてコードを隠す（任意）: - セルの右上メニュー（縦三点リーダー）から「フォーム」➔「コードを非表示」を選択すると、一般利用者の画面には生コードが隠れ、タイトルと入力フォーム（実行ボタン）だけが表示された綺麗な UI として配布できます。 4. 後続セル（Step 2）から実行: - 次のセル（Step 2）を作成し、from {PACKAGE_NAME}.main import run_pipeline; run_pipeline()（例: from src.main import run_pipeline; run_pipeline()）のように呼び出せば、クローンされたコードがそのまま動作します。 :::note info パッケージ名（ここでは src）は、サードパーティ製ライブラリと重複しない固有のパッケージ名（例: my_tool）にしておくと、名前空間の衝突をより確実に防ぐことができます。 ::: 2. コラム：プライベートリポジトリを安全に扱うには？ もしコードを一般公開せず、チームメンバーや特定利用者向けに配布する場合、トークンをハードコードしてはいけません。Colab 標準の 「Secrets（🔑）」機能 を使って環境変数経由でトークンを注入し、クローン完了後にローカルの Git 設定からトークンを抹消します。 # プライベートリポジトリの場合の安全なトークン注入と痕跡消去 import re import subprocess from google.colab import userdata try: gh_token = userdata.get("GITHUB_TOKEN") auth_repo_url = f"https://x-access-token:{gh_token}@github.com/my-org/private-tool.git" except Exception: raise RuntimeError("🔑 左メニューの鍵アイコン (Secrets) から 'GITHUB_TOKEN' を設定してください。") # クローン実行（エラー出力時にトークンが露出しないよう URL 部分を伏字化） try: subprocess.run( ["git", "clone", "--depth", "1", "--branch", TARGET_TAG, auth_repo_url, TARGET_DIR], check=True, capture_output=True, text=True, ) # クローン成功後、ローカル .git/config 内の URL からトークンを即時除去 clean_repo_url = "https://github.com/my-org/private-tool.git" subprocess.run( ["git", "-C", TARGET_DIR, "remote", "set-url", "origin", clean_repo_url], check=True, ) except subprocess.CalledProcessError as e: raw_msg = e.stderr.strip() if e.stderr else "認証またはリポジトリの取得に失敗しました" err_msg = re.sub(r"https://[^@]+@", "https://***@", raw_msg) raise RuntimeError(f"❌ プライベートリポジトリの取得エラー: {err_msg}") from None --- 第6章：動かしてみた結果とベンチマーク このセットアップ処理の導入により、運用の安定性が向上しました。 1. 起動時間とエラー耐性の比較 数百コミット規模のリポジトリを Colab 無料枠でクローンした筆者環境での実測値です。 | 項目 / 状態 | 通常の !git clone | 改善したセットアップ | 改善効果 | | :--------------------------- | :---------------------------: | :----------------------------------: | :------------------------- | | 初動の転送時間 | 約 5〜10 秒（全履歴取得時） | 約 2〜4 秒（--depth 1） | 初動の短縮 | | 同一セルの再実行 | already exists でエラー停止 | 自動消去・再取得で衝突なし | 冪等性の担保 | | タグ push 漏れ・タイポ時 | エラーによる処理中断 | 検証を経て許可時のみ main 退避 | タグ不在時に main 代替取得 | | 自作モジュール更新 | 再起動しないと旧コード参照 | sys.modules 破棄で即反映 | コード不整合の防止 | 2. 同一ランタイムでのセル再実行に対する冪等性と復旧 セッションが完全に切断された場合は Colab の VM ごと初期化されるため前回の残骸は残りませんが、実際の運用で頻繁に発生するのは「エラー発生後やパラメータ変更時に、同一ランタイム上でセルをもう一度実行する」ケースです。 セットアップコードの冒頭でカレントディレクトリを /content に戻し、既存フォルダを消去することで、同じセッション内でのやり直し時にディレクトリの衝突や、削除済みフォルダ内に留まって迷子になる問題を防ぎます。 ※なお、pip install の実行途中で処理を強制中断した場合、site-packages 配下に不完全なパッケージが残存する可能性があります。その場合はノートブックのメニューからランタイムの再起動（Restart session）を行うか、pip install --force-reinstall を実行して修復します。 3. 今回の設計が持つトレードオフと運用上の制約 本設計は「一般利用者の手元で止まりにくく、安全に再実行できること」を優先しています。そのため、以下のトレードオフ（何を得て、何を妥協したか）が存在します。 - 【起動速度 vs Git履歴の閲覧性（--depth 1 の代償）】最新コミットに絞ることで履歴追跡は切り捨てる: --depth 1 によって数秒での高速クローンを実現した代償として、過去のコミット履歴（git log や git diff、過去リビジョンへのチェックアウト）は利用できません。コードを配布して実行させる用途に特化し、開発者向けの履歴閲覧性は意図して捨てています - 【可用性（止まらないこと） vs コードの再現性】main への代替切り替えは未検証コードのリスクを伴う: タグ不在時に main ブランチへ自動退避して処理を継続させる選択肢を用意した代償として、開発途中の未検証コードが動いてしまうリスクを負います。そのため本設計では ALLOW_MAIN_FALLBACK をデフォルトで False とし、利用者がリスクを承知の上で明示的に有効化した場合のみ main へ切り替える安全弁を設けています - 【高速な再読込 vs C拡張モジュールの更新限界】インメモリ破棄で扱えるのは純粋な Python コードに限定される: ランタイム再起動を挟まずに sys.modules の破棄で最新コードを即時反映させる軽快さを取った代償として、numpy や torch など一度メモリに共有ライブラリ（.so）がロードされたバイナリ拡張パッケージの更新は反映できません。これらをアップグレードした場合は、OS プロセス自体の再起動（ランタイム再起動）が必要です - 【安全なトークン注入 vs ログ出力・プロセス一覧からの漏洩リスク】例外メッセージのサニタイズが必要になる: Secrets を用いてプライベートリポジトリを安全にクローンできる利便性を取った代償として、例外発生時にトークンが露出しないよう stderr のサニタイズ（伏字化）が必要です。また、クローン実行中は OS のプロセス一覧（ps）から一時的に引数のトークンが見える点にも留意が必要です --- 結び Google Colab 上でリポジトリのコードを動かす作業は、ローカル開発環境での git clone とは前提が異なります。 「作業ディレクトリを /content に戻す」「既存フォルダの事前クリーンアップ」「--depth 1 による最新取得」「タグ不在時の main 代替取得」「モジュールキャッシュの破棄」という一連の対策をセットアップセルに組み込むことで、ボタン1つで安定して起動できる配布基盤を構築できます。 本稿で確立したコード展開フローと、第1弾（データ永続化編）で設計した 2 層ストレージフローを組み合わせることで、ノートブック全体のライフサイクルは以下のように整理されます。 > 【Step 1】環境セットアップ（本稿：コード展開・依存解決） > ↓ > 【Step 2】Pull Phase（第1弾：永続層 Drive から作業層ローカルディスクへデータステージング展開） > ↓ > 【Step 3】処理実行（ローカルディスク上での高速バッチ・クエリ処理） > ↓ > 【Step 4】Push Phase（第1弾：成果物先行・DB末尾置換でデータステージング書き戻し ＆ アンマウント） 2 つの設計を組み合わせることで、対話的なノートブックであっても運用に耐えうる安定したバッチ実行基盤として活用できるようになります。 --- > 免責事項（Disclaimer） > 本記事に掲載されているコードや設定例は筆者の検証環境に基づくものであり、外部サービス（Google Colab、GitHub 等）の仕様変更や実行環境の違い等によって生じたいかなる損害についても責任を負いかねます。実際の運用や設定はご自身の責任において確認の上で行ってください。