googleapis

OpenTelemetry Collector Custom Builds: 6.5x Smaller, and What Resource Detectors Cost at Startup --- title: OpenTelemetry Collector Custom Builds: 6.5x Smaller, and What Resource Detectors Cost at Startup published: true date: 2026-11-05 15:00:00 UTC tags: DevOps,OpenTelemetry,Observability,Go canonical_url: devops-daily.com --- The contrib release of the OpenTelemetry Collector 0.164.0 is a 420.3 MB binary on linux/amd64. We built a Collector with the OpenTelemetry Collector Builder (ocb) that has only the 10 components a VM agent uses and only the env and system resource detectors. It is 64.3 MB, so contrib is 6.5 times larger. After 10 seconds idle, the custom build held 45.1 MiB of resident memory and contrib held 199.8 MiB. The anonymous part (heap, stacks and runtime data) was 8.0 MiB against 34.8 MiB. The binary adds a little to startup: contrib needed 110 to 160 ms more than the small build on the droplet, and 300 to 500 ms more on the Pi. The detector list in your config can add seconds. We started both builds with cloud detectors for clouds the host was not on, in four network setups. In every list without akamai, each start waited for the slowest detector. When that detector got no answer, it waited for the whole timeout, which is 5 s by default. That happened on a Raspberry Pi with no metadata service. With a list of 13 cloud detectors it also happened on a DigitalOcean droplet, because two of them ask addresses that DigitalOcean does not answer. The detectors run in parallel, so where nothing answered, the list with 3 cloud detectors and the list with 13 cost the same 5 s. timeout: 1s cut the wait to about 1 s in most cases. Two detectors do not obey it. digitalocean ignores the deadline, so where nothing answers it waits its own 2 s. akamai makes its network call while the processor is created. In all four setups we tested, none of them on Akamai, the Collector did not start. It exited at once where the request got an answer or a fast error, and after 30 s where it got no answer. On 29 October the OpenTelemetry blog showed how to compile detectors out with ocb and reported a Kubernetes distribution about 20% smaller. This post measures three things that page does not cover. What does the same filter do to a small build? What does the build do to memory at idle? And what does the detector list cost at startup on hosts that are not on the clouds the config names? Every number below comes from a file in the companion repo. TL;DR - Size. On amd64, contrib is 420.3 MB, the core otelcol release is 183.8 MB, our agent with every detector is 124.1 MB, and the same agent with only env and system is 64.3 MB. On this small build, the detector filter alone removed 48% of the binary and 64 Go modules: the AWS SDK, the Google Cloud clients, the Docker client, the Consul API and four more cloud metadata clients. arm64 gives the same picture. - Memory. After 10 s idle: contrib 199.8 MiB RSS, core 98.4 MiB, our agent 61.5 MiB, the agent without cloud detectors 45.1 MiB. About four fifths of each number is file-backed memory (RssFile), which grows with the binary. The anonymous part (RssAnon) was 34.8 MiB for contrib and 8.0 MiB for the small build. - Startup with no cloud detectors: about 190 to 220 ms for contrib and 45 to 85 ms for the small agents on the droplet, and 480 to 610 ms against 110 to 270 ms on the Pi. - Startup with cloud detectors for other clouds: [env, system, ec2, gcp, azure] held startup for the full 5 s timeout where 169.254.169.254 got no answer. Where the connect failed at once, it held startup for 4.1 s, because the AWS SDK retries the connect. On the droplet those three cost under 4 ms each, but the 13-cloud list still took 5.1 s there, because alibaba_ecs and scaleway ask other addresses. - The timeout setting bounds the wait: 1 s, 5 s or 10 s plus normal startup. The exceptions are digitalocean, which waits its own 2 s whatever you set when nothing answers, and akamai. - No start at all. akamai in the list: exit 1 in all four setups, after 30 s in the two with no answer. fail_on_missing_metadata: true: exit 1 after the 5 s timeout. A detector that ocb compiled out but the config still names: validate passes, then a nil pointer panic at start. The processor's name for a detector in the ocb manifest (ec2 instead of aws_ec2): the build stops. - What to do. List only the detectors for the platform the host runs on, and build with the same list in dist.resource.detection.detectors. If one config must cover several clouds, keep the list short, set timeout to 1 or 2 s, and leave out akamai. Prerequisites - An x86_64 Linux host for the builds, with about 3 GB of free disk. ocb cross-compiles the arm64 binaries there. - curl, python3 and ip (iproute2). Root only for the two network namespaces. - An arm64 Linux host if you want the second architecture. We used a Raspberry Pi with 4 cores and less than 4 GB of memory. - About an hour for a full run. The cases that fail after 30 s take most of it. What we built and where we ran it We downloaded the official 0.164.0 releases of otelcol-contrib and otelcol for both architectures and checked each tarball against its published .sha256. Then ocb 0.164.0 built two versions of one small agent. It has the components a VM agent uses: - receivers: OTLP, hostmetrics and filelog - processors: batch, memory_limiter and resource_detection - exporters: debug, OTLP and OTLP/HTTP - extension: health_check otelcol-agent uses the ocb default, so every detector is compiled in. otelcol-agent-envsys adds the ocb setting that compiles out every detector you do not list: # manifests/agent-envsys.yaml (excerpt, comment on detectors added) dist: name: otelcol-agent-envsys version: 0.164.0 build_tags: 'grpcnotrace' # same tag as the contrib release manifest resource: detection: detectors: # ocb 0.163.1 and later: every detector not listed gets an omit_detector_<name> build tag - env: - system: go version -m on the release binaries reports go1.26.10, so we pinned the same Go for our builds. go version -m and file show -trimpath=true, CGO_ENABLED=0 and stripped symbols for every binary (results/buildinfo/). Every start ran the same pipeline: OTLP in, batch, debug out. The harness gave each start a clean environment with only PATH and an empty HOME. No OTEL_* or AWS_* variable and no ~/.aws file could change what a detector found. The Pi runs also set TZ=UTC, so their logs use UTC like the droplet's; the droplet runs used an earlier revision of the harness without it. Only the resource_detection block changed between configs. Each case ran 5 times in a shuffled order, after one warm-up start per binary that we left out of the numbers. - DigitalOcean droplet 4 vCPU, 8 GB, amd64 - host network metadata server answers - blackhole namespace packets dropped - noroute namespace connect fails at once - Raspberry Pi 4 cores, under 4 GB, arm64 - home network router drops 169.254.169.254 The four network setups matter more than the two architectures: | Setup | What a request to 169.254.169.254 gets | | --- | --- | | Droplet host | DigitalOcean's metadata server answers. The EC2 and Azure paths return 404 in about 1 ms. This is the "host on another cloud" case. | | Blackhole namespace | The default route goes to a dummy device, which discards every packet, so a connect gets no answer. DNS fails at once, because nothing listens on 127.0.0.53 inside the namespace. | | Noroute namespace | Loopback only. A connect fails at once with "network is unreachable". | | Raspberry Pi | The home router drops the packets. DNS works. This is the plain "not on any cloud" host. | The-DevOps-Daily/otelcol-ocb-vs-contrib on GitHub Size: what ocb removes Binary size, linux/amd64, OpenTelemetry Collector 0.164.0 | | Value | Series | | --- | --- | --- | | otelcol-contrib (release) | 420.3 MB | release | | otelcol core (release) | 183.8 MB | release | | agent, all detectors | 124.1 MB | custom | | agent, env + system only | 64.3 MB | custom | _results/sizes.tsv. Stripped, -trimpath, CGO_ENABLED=0, go1.26.10. The two agents have the same 10 components; only the compiled-in detectors differ._ The contrib manifest lists 246 components and 9 config providers, and the binary links 1,088 Go modules. The core release is not small either. Its 30 components include the Kafka and Prometheus receivers and exporters, and it also ships resource_detection with every detector. Our agent with every detector links 289 modules, and the detector filter takes it to 225. The filter saves more on a small build than on a big one. The OpenTelemetry blog measured the Kubernetes distribution and got 182 MB to 147 MB with 13 fewer dependencies. On our agent it removed 59.8 MB of 124.1 MB (48%) on amd64, 52.9 MB of 112.7 MB (47%) on arm64, and 64 modules. A big distribution needs most of those SDKs for other components, so removing the detectors does not remove the SDKs. In a small agent, nothing else needs them. results/deps/removed-by-detector-filter-amd64.txt lists the 64: 15 aws-sdk-go-v2 and smithy-go modules, the Google Cloud compute, auth and API clients, the Docker (moby) client, the Consul API with its HashiCorp libraries, and the Hetzner, Scaleway, Linode and DigitalOcean clients. For containers, the official otel/opentelemetry-collector-contrib:0.164.0 image is 116.1 MB compressed on amd64 (results/images.tsv). We did not build an image of the small agent. Its binary compresses to 20.3 MB with gzip -6, so a FROM scratch image would be near that, but that is an estimate, not a measurement. One build difference favours our side. The contrib manifest asks for the grpcnotrace build tag and our builds use it, but the release binaries record no build tags in go version -m. We built otelcol-agent-notag to put a size on it: 124.23 MB against 124.11 MB, so the tag is worth 0.1%. Memory at idle Resident memory after 10 s idle, droplet, amd64 | | Value | Series | | --- | --- | --- | | contrib: RssAnon | 34.8 MiB | RssAnon | | contrib: RssFile | 165.2 MiB | RssFile | | core: RssAnon | 18.7 MiB | RssAnon | | core: RssFile | 79.5 MiB | RssFile | | agent, all detectors: RssAnon | 11.2 MiB | RssAnon | | agent, all detectors: RssFile | 50.2 MiB | RssFile | | agent, env + system: RssAnon | 8 MiB | RssAnon | | agent, env + system: RssFile | 37 MiB | RssFile | _Median of 5 starts, /proc//status, config without resource_detection, results/summary.md. RssFile is resident file-backed memory; RssAnon is resident anonymous memory (heap, stacks, runtime)._ Resident memory follows the binary size, but not one to one. Contrib held 199.8 MiB, 4.4 times the 45.1 MiB of the small agent. About four fifths of every number is RssFile, resident memory backed by files. It grows with the binary: 165.2 MiB for contrib, 37.0 MiB for the small agent. That points at the binary's own pages, but we did not map them one by one. The kernel can drop clean file-backed pages under memory pressure and read them again later. RssAnon is anonymous memory, which it cannot drop without swap. There contrib held 34.8 MiB against 8.0 MiB. A Go program runs the init code of every package it links before main, so a likely part of that difference is setup work for code that this pipeline never uses. We did not profile the heap to prove it. Detection with [env, system] added little: contrib went to 205.4 MiB and the small agent to 47.8 MiB. The Raspberry Pi gave the same order on arm64: contrib 183.8 MiB with 34.7 MiB anonymous, the small agent 40.6 MiB with 8.0 MiB. We did not test what happens to RssFile under memory pressure, or how a cgroup memory limit counts it. Read the RssAnon column as the floor and VmRSS as what top shows you. Startup: the slowest detector sets the time Median time from exec to the log line "Everything is ready", 5 starts each, from results/summary.md. Each cell is the small agent with every detector, then contrib in brackets: | Config | Droplet host | Blackhole namespace | Noroute namespace | Raspberry Pi | | --- | --- | --- | --- | --- | | [env, system] | 79 ms (189 ms) | 77 ms (210 ms) | 83 ms (221 ms) | 192 ms (481 ms) | | + ec2, gcp, azure | 81 ms (196 ms) | 5,088 ms (5,203 ms) | 4,098 ms (4,242 ms) | 5,172 ms (5,486 ms) | | + 13 VM clouds, no akamai | 5,090 ms (5,213 ms) | 5,086 ms (5,203 ms) | 4,087 ms (4,227 ms) | 5,167 ms (5,393 ms) | | + akamai | exit 1 after 0.1 s (0.2 s) | exit 1 after 30.1 s (30.2 s) | exit 1 after 0.1 s (0.2 s) | exit 1 after 30.2 s (30.5 s) | Without a network wait, contrib needs about 110 to 160 ms more than the small agent to start on the droplet, and about 300 to 500 ms more on the Pi. That is real but small, and it matters only if you start collectors very often, for example one per short CI job. The wait comes from three things in the processor source (processor/resourcedetectionprocessor at v0.164.1): 1. Detectors run in parallel. detectResource starts one goroutine per detector and then reads their results in order (internal/resourcedetection.go, lines 203 to 222). The wait is the slowest detector, not the sum. 2. The whole detection has one deadline: timeout. The default is 5 s (factory.go, lines 153 to 157). With the default retry.max_elapsed_time: 0, Refresh puts that timeout on the context that every detector gets (internal/resourcedetection.go, lines 159 to 168). 3. Startup waits for it. The processor's Start runs the first detection and returns its error (resourcedetection_processor.go, lines 36 to 40). The Collector logs "Everything is ready" only after every processor has started. The processor also records how long each detector took, in its own otelcol_resourcedetection_detector_duration histogram. We read it from the Collector's Prometheus endpoint after each start. This is the 13-cloud list in the blackhole namespace: Time per detector, 13 VM cloud detectors, blackhole namespace, timeout 5s | | Value | Series | | --- | --- | --- | | ec2 | 5000.7 ms | waits for timeout | | alibaba_ecs | 4999.8 ms | waits for timeout | | azure | 4999.7 ms | waits for timeout | | hetzner | 5000.2 ms | waits for timeout | | scaleway | 4999.6 ms | waits for timeout | | digitalocean | 2000.6 ms | own 2 s timeout | | nova | 2000.4 ms | own 2 s timeout | | upcloud | 2000.7 ms | own 2 s timeout | | vultr | 2000.8 ms | own 2 s timeout | | oraclecloud | 200.7 ms | fast | | ibmcloud_vpc | 1.2 ms | fast | | gcp | 0.9 ms | fast | | tencent_cvm | 0.7 ms | fast | _Median of 5 starts of the small agent with all detectors, from otelcol_resourcedetection_detector_duration in results/runs/droplet-blackhole-extra.jsonl. Every detector reported outcome=success: "not on this cloud" is a success with no attributes._ The same histogram tells you which detector held each of the other setups: | Setup | Detectors that took the longest | | --- | --- | | Droplet host | alibaba_ecs and scaleway, 5.0 s each. They ask 100.100.100.200 and 169.254.42.42, and nothing on DigitalOcean answers those. The other 11 finished in 2 to 20 ms (medians). | | Blackhole namespace | Five detectors at 5.0 s and four at 2.0 s (see the chart). | | Noroute namespace | ec2 alone, 4.0 s. Every other detector finished in under 2 ms. | | Raspberry Pi | Six detectors at 5.0 s: the five from the blackhole namespace and ibmcloud_vpc, which took 1.2 ms in the namespace. We did not trace why. The same four at 2.0 s. With working DNS, gcp took 13.5 ms and tencent_cvm 15.9 ms. | How each detector probes decides how long it waits: - azure sends one GET with a plain &http.Client{} that has no timeout (internal/metadataproviders/azure/metadata.go, lines 33 to 38). When the packets get no answer, it waits for the processor's deadline. Then it returns an empty resource and no error (internal/azure/azure.go, lines 55 to 63). - ec2 uses the AWS SDK's IMDS client: a 250 ms dial timeout, a 500 ms response header timeout (aws-sdk-go-v2 feature/ec2/imds v1.20.1, api_client.go, lines 277 and 278), and up to 3 attempts per request (internal/aws/ec2/config.go, lines 16 to 19). A connect that fails at once is retried with backoff, which is why ec2 took 4.0 s in the noroute namespace. With no answer at all and a 10 s timeout, it gave up by itself after 5.5 s. - gcp calls metadata.OnGCEWithContext, which races a DNS lookup of metadata.google.internal against an HTTP probe. On a host that does not look like a Google VM, it trusts the first answer (cloud.google.com/go/compute/metadata v0.10.0, metadata.go, lines 456 to 480). A fast NXDOMAIN or a fast DNS failure ends it. - digitalocean ignores the context: its Detect takes _ context.Context (internal/digitalocean/digitalocean.go, line 51), and the DigitalOcean metadata client has a fixed 2 s timeout (digitalocean/go-metadata, client.go, lines 25 and 64). We started from this claim: on a host that is not on EC2, Google Cloud or Azure, each cloud detector adds a wait at startup. Two results would prove it wrong: no wait at all, or the same wait for 3 detectors and for 13. We got the second, so it was half wrong. The detectors do not each add a wait. During detection they overlap, and the slowest one sets the time. But a host on a cloud with a metadata server is not safe either, because not every cloud uses 169.254.169.254. The timeout setting Start to "Everything is ready" with [env, system, ec2, gcp, azure] | | timeout: 1s | timeout: 5s (default) | timeout: 10s | | --- | --- | --- | --- | | contrib, blackhole namespace | 1238.8 ms | 5203.1 ms | 10235.4 ms | | agent, blackhole namespace | 1077.1 ms | 5087.6 ms | 10085.4 ms | | contrib, Raspberry Pi | 1526.7 ms | 5485.5 ms | 10710.8 ms | | contrib, noroute namespace | 1240.1 ms | 4241.6 ms | 4229.5 ms | | contrib, droplet host | 209.3 ms | 196 ms | 216 ms | _Median of 5 starts per point, results/summary.md. The droplet host answers on 169.254.169.254; the blackhole namespace and the Pi do not. The noroute namespace fails the connect at once._ Where nothing answers, the wait is the timeout plus normal startup, on both architectures. In the noroute namespace it stops at 4.1 to 4.2 s, where ec2 gives up. timeout: 1s brought every three-cloud case that waited down to 1.1 to 1.2 s. The 13-cloud list with timeout: 1s did not get there. In the blackhole namespace it started in 2,097 ms (contrib 2,234 ms), and on the Pi in 2,175 ms (contrib 2,410 ms), because digitalocean took its 2 s anyway. A short timeout has a cost that we could not measure. On a real EC2, Google Cloud or Azure VM, a detector that needs more than 1 s gets cut off. It returns an empty resource with no error, and your telemetry has no cloud attributes. We did not run on those clouds, so we cannot say whether 1 s is enough there. Most examples in the processor README use timeout: 2s. Four ways the Collector does not start 1. ocb build unknown detector name: stops 2. otelcol validate passes a compiled-out detector 3. create processor akamai fetches a token 4. Start: detection parallel, one deadline 5. Everything is ready 1. The akamai detector Of the 14 VM cloud detectors we ran, akamai is the only one that waited or failed while the processor was created. Its NewDetector calls newAkamaiClient(context.Background()) (internal/akamai/akamai.go, lines 43 to 57), and the Linode metadata client fetches a token inside NewClient (linode/go-metadata v0.3.1, client.go, lines 88 to 91). This happens before detection, so timeout does not apply, and a failure there is a failure to build the pipeline. On the droplet, DigitalOcean's metadata server answers the token request, but not with a token, and the Collector exits at once: results/runs/droplet-host-extra.jsonl, otelcol-contrib, configs/akamai.yaml $ bin/linux-amd64/otelcol-contrib --config configs/akamai.yaml Error: failed to build pipelines: failed to create "resource_detection" processor, in pipeline "traces": failed creating detector type "akamai": failed to refresh metadata token: failed to generate metadata token: no token returned from API # exit code 1, 0.2 s after exec In the noroute namespace it exited at once with connect: network is unreachable. In the blackhole namespace and on the Pi, the token request first waited for a 30 s dial timeout. The Collector exited after 30.1 s in the namespace and 30.2 s on the Pi (contrib: 30.2 s and 30.5 s). With timeout: 1s it still took 30 s. 2. fail_on_missing_metadata: true This setting makes a cloud detector return an error when its metadata service is missing, instead of an empty resource. Retry is on by default (factory.go, lines 142 to 151), so the processor retries the detector with backoff until timeout, and then Start returns the error. With [env, system, ec2, gcp, azure], the Collector exited with code 1 after 5.1 to 5.3 s in all three droplet setups. The config comment says the setting is for nodes that boot before their metadata service is ready (config.go, lines 63 to 70). It also turns "not on EC2" into "does not start". 3. A detector that ocb compiled out This is the trap in the new ocb feature. otelcol-agent-envsys has only env and system. We gave it the three-cloud config: results/break/amd64-validate.txt and results/break/amd64-start.txt $ bin/linux-amd64/otelcol-agent-envsys validate --config configs/clouds3.yaml exit code: 0 $ bin/linux-amd64/otelcol-agent-envsys --config configs/clouds3.yaml panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x130a757] goroutine 57 [running]: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.(*ResourceProvider).detectWithRetry.func2() github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.164.1/internal/resourcedetection.go:314 +0xb7 ... exit code: 2 In v0.164.1 each *_omit.go stub returns nil, nil from NewDetector, for example internal/aws/ec2/ec2_omit.go. The processor keeps the nil detector and calls it. This is issue #52519, open when we wrote this. On main, #52063 changes the stubs so they register no factory. It was merged after v0.164.1. With it, the processor returns detector "ec2" is not compiled into this binary with the list of compiled detectors. Until a release has that change, validate does not catch the mismatch, and the first sign is a crash. 4. The processor's names in the ocb manifest ocb and the processor do not use the same detector names. The processor config says ec2, nova and aks. The ocb manifest wants the build tag names aws_ec2, openstack_nova and azure_aks (cmd/builder/internal/builder/config.go, lines 117 to 151). This one fails early, which is the good kind of failure: results/build/agent-badname-amd64.log $ ocb --config manifests/agent-badname.yaml Error: invalid configuration: ec2: unknown resource detector What to put in the manifest and the config Make the two lists match, each in its own names: # ocb manifest: build tag names dist: resource: detection: detectors: - env: - system: - aws_ec2: # the processor calls this one "ec2" # Collector config: processor names, the same detectors processors: resource_detection: detectors: [env, system, ec2] timeout: 2s # the most a missing metadata service can add to startup - Use one list per platform. A fleet on EC2 does not need azure, and a fleet on bare metal needs neither. - If one config must cover several clouds, keep the list short and set timeout to 1 or 2 s. Expect digitalocean to add 2 s, whatever the timeout, on hosts where its request gets no answer. - Leave akamai out unless the hosts run on Akamai Cloud. - Keep fail_on_missing_metadata off unless "no metadata" should mean "do not start". - Use the name resource_detection. In 0.164 the old name resourcedetection still works, with a deprecation warning (results/break/amd64-legacy-name.txt). - Until the #52063 change is in a release, start each new custom build once with its real config before you ship it. validate is not enough. What we could not conclude - On a real AWS, Google Cloud or Azure VM. We had no instances there, so we did not measure how long a detector takes when it succeeds, or whether a 1 s timeout cuts it off. - The container image of the small agent. We report the compressed size of the official image and the gzip size of our binary. We did not build or push an image. - Memory under pressure. RssFile can be reclaimed. We did not measure how much the kernel takes back, or how a cgroup limit counts it. - Small startup differences. Each case has 5 starts. The 110 to 160 ms between contrib and the small agent repeats in every droplet setup. The Pi also ran other work, and three of its starts took 4.6 to 22 s. In all three, the time went before the Collector wrote its first log line, which points at reading the binary from the SD card, not at detection. We did not confirm that. They move the medians little, but they are in the min and max columns of results/summary.md. - Slow DNS. DNS failed at once in the namespaces and answered on the Pi, so gcp and tencent_cvm were fast everywhere. A host whose DNS times out would make them wait too. We did not test that. - Other versions. Everything here is 0.164.0 and 0.164.1. The akamai and omit behaviour may change in the next release. Related reading: - Distributed tracing with OpenTelemetry runs a Collector end to end. - OpenTelemetry graduated covers which vendor agents a Collector can replace. - Trace context in environment variables follows a trace through a CI pipeline. Summary A custom Collector with only the components you use is 6.5 times smaller than contrib and holds under a quarter of its memory. On a small build, compiling out the unused detectors is half of the size saving. The binary costs tens to hundreds of milliseconds at startup. The detector list can cost seconds. Apart from akamai, the cloud detectors you list for clouds the host is not on share one wait: the slowest probe. Where that probe gets no answer, it usually runs to the full timeout, even on a host on another cloud. digitalocean waits its own 2 s instead, even past a shorter timeout. Keep the list to the platforms you run on, and build with the same list. Set a short timeout if the list must stay long. Start each new binary once with its real config. Leave akamai out unless you run on Akamai. _Originally published at devops-daily.com._

How to Get YouTube Video Stats With Python and the YouTube Data API If you run a YouTube channel, or build tools for people who do, sooner or later you want the numbers in a spreadsheet instead of clicking through YouTube Studio one video at a time. The YouTube Data API v3 gives you public stats (views, likes, comment counts, titles, publish dates) for any public video, and it is free within a daily quota. This post walks through a small Python script that lists every upload on a channel and saves its stats to a CSV file. Note: this guide was written with the help of AI tools and is based on the official YouTube Data API documentation. Check the docs linked at the end for the latest details. What you can and cannot get The Data API returns public information. For each video that means: - title, description, tags and publish date (snippet) - view count, like count and comment count (statistics) - duration and definition (contentDetails) It does not return private analytics such as watch time, audience retention, click through rate or revenue. Those live in the separate YouTube Analytics API, which needs OAuth and the channel owner's permission. For many dashboards and research scripts, the public numbers are enough. Step 1: Create an API key 1. Open the Google Cloud Console and create a project (or pick an existing one). 2. Go to APIs and Services, then Library, search for YouTube Data API v3 and enable it. 3. Go to Credentials, click Create credentials, then API key. 4. Restrict the key to the YouTube Data API v3 so it is less useful to anyone who finds it. Keep the key out of your code. An environment variable is the simplest option: export YT_API_KEY="paste-your-key-here" Step 2: Install the client library pip install google-api-python-client Step 3: Find the channel's uploads playlist Every channel has a hidden playlist that contains all of its public uploads. Reading that playlist is far cheaper than using the search endpoint, so it is the right way to list videos. import os from googleapiclient.discovery import build API_KEY = os.environ["YT_API_KEY"] youtube = build("youtube", "v3", developerKey=API_KEY) def get_uploads_playlist(channel_id): resp = youtube.channels().list( part="contentDetails", id=channel_id, ).execute() return resp["items"][0]["contentDetails"]["relatedPlaylists"]["uploads"] If you only know the channel's handle (the @name part of the URL), recent versions of the API accept forHandle="@name" in place of id=channel_id. Step 4: Collect every video ID playlistItems.list returns at most 50 items per call, so loop with pageToken until there are no more pages. def get_video_ids(playlist_id): ids, token = [], None while True: resp = youtube.playlistItems().list( part="contentDetails", playlistId=playlist_id, maxResults=50, pageToken=token, ).execute() ids += [item["contentDetails"]["videoId"] for item in resp["items"]] token = resp.get("nextPageToken") if not token: return ids Step 5: Fetch stats in batches of 50 videos.list accepts up to 50 comma separated IDs per request, which keeps the number of calls small. def get_stats(video_ids): rows = [] for i in range(0, len(video_ids), 50): batch = video_ids[i:i + 50] resp = youtube.videos().list( part="snippet,statistics,contentDetails", id=",".join(batch), ).execute() for v in resp["items"]: s = v.get("statistics", {}) rows.append({ "video_id": v["id"], "title": v["snippet"]["title"], "published": v["snippet"]["publishedAt"], "duration": v["contentDetails"]["duration"], "views": int(s.get("viewCount", 0)), "likes": int(s.get("likeCount", 0)), "comments": int(s.get("commentCount", 0)), }) return rows Two details worth knowing: - likeCount and commentCount can be missing when a creator hides likes or turns comments off, which is why the code uses .get() with a default. - duration comes back in ISO 8601 format, such as PT4M13S. The isodate package can turn it into seconds if you need to separate Shorts from long videos. Step 6: Save to CSV import csv def save_csv(rows, path="channel_stats.csv"): with open(path, "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=rows[0].keys()) writer.writeheader() writer.writerows(rows) if __name__ == "__main__": channel_id = "UC_your_channel_id_here" playlist = get_uploads_playlist(channel_id) ids = get_video_ids(playlist) rows = get_stats(ids) rows.sort(key=lambda r: r["views"], reverse=True) save_csv(rows) print(f"Saved {len(rows)} videos") Open the CSV in Excel or Google Sheets and you can sort by views, compare titles, or chart performance over time. Mind the quota New projects get a default quota of 10,000 units per day. Each channels.list, playlistItems.list and videos.list call costs 1 unit, so even a channel with a few thousand videos only uses a few hundred units with this approach. The search.list endpoint costs 100 units per call, which is why this script avoids it. If you search for videos by keyword, a hundred searches will use the whole day's quota. Ideas to take it further - Run the script daily (cron or a scheduled GitHub Action) and store each run with a date, so you can see how views grow after upload. - Compare your channel with a few channels in the same niche to see which topics and title styles get the most views. - Split Shorts from long videos by duration and look at them separately, since they behave very differently. If you would rather not write code, creator tools such as vidIQ and TubeBuddy show many of these public stats inside YouTube for you. But if you want the raw numbers in your own spreadsheet or app, a short script like this does the job. Useful links - YouTube Data API overview: developers.google.com - Quota costs: developers.google.com - Python client library: googleapis/google-api-python-client

Upgrade a Major Dependency With a Coding Agent: Changelog First, Then the Failing Tests - Book: AI That Acts - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- openai-node 7.0.0 came out on July 27, 2026, with one item under breaking changes: the SDK now requires Node.js 22. Three months later, 7.44.0, a minor release, carries the line "Breaking change in a minor release." It says callers that directly access item.content or item.role on Live Decision items "may no longer compile." 7.43.0, a few days earlier, has the same label for a change to DecisionInputMessage. Type "upgrade openai to v7" into a coding agent. It bumps the version and starts reading type errors. It has never seen those release notes. Every fix it makes is a guess about what changed, and in a diff a guess that compiles looks just like a fix that follows the release note. The order I'd use: find out what the new version demands, read what changed in every release you are skipping, record what actually broke in your code. Then the agent gets all three as its brief. Steps 1 to 3 are plain gh, npm, git and tsc in a terminal. From step 4 on I use Hermes IDE, which I build, so weigh that part accordingly. The running example is a TypeScript app in ~/code/app on openai@6.49.0, the release that the 7.0.0 notes compare against, going to 7.47.0, the latest release as of October 31. Swap in your own package and versions. Step 1: Ask npm what the new version demands Four read-only commands in your main checkout. Nothing changes on disk. cd ~/code/app npm outdated openai npm view openai@7.47.0 engines peerDependencies node --version npm explain zod npm outdated shows Current, Wanted and Latest. Wanted is the highest version your package.json range allows, so with ^6.49.0 it stays on 6.x while Latest shows 7.x. npm prints that row in yellow, its colour for a newer version outside your range. npm view reads the target's manifest from the registry before you install anything. The 7.47.0 package.json sets "node": ">=22.0.0" and lists optional peers, among them zod at ^3.25 || ^4.0 and ws at ^8.21.0. Both fields bite later than you'd think. npm's package.json docs say that unless you set engine-strict, the engines field "will only produce warnings." On Node 20 the install goes through and you find out in CI or at runtime. Peers bite only if you already have them. openai marks zod and ws optional, and the same docs say "Npm will not automatically install optional peer dependencies." But if your tree already holds a zod or ws outside those ranges, a conflicting requirement "may cause an error if the tree cannot be resolved correctly." npm explain zod prints the chain of dependencies behind each copy of zod you have, so you know whether your code or some other package owns it. If node --version prints anything below 22, that is its own change. Make it before the SDK bump, or at least name it in the brief. Step 2: Pull every release note in the range First, keep all the working files out of git. Every file in this tutorial starts with .upgrade-: echo '.upgrade-*' >> .gitignore The major release's own notes are one call to gh release view: gh release view v7.0.0 -R openai/openai-node \ --json body -q .body Those are the notes most people read, and for this upgrade they say the least. To get every release between your version and the target, list them all with gh api and filter them with jq. Save this filter as .upgrade-between.jq: .[][] | select(.draft | not) | select(.tag_name | test("^v[0-9]+\\.[0-9]+\\.[0-9]+$")) | (.tag_name | ltrimstr("v") | split(".") | map(tonumber)) as $v | select($v > $from and $v <= $to) | "## \(.tag_name) (\(.published_at[0:10]))\n\n\(.body // "")\n" Then fetch, filter and search: gh api --paginate --slurp \ 'repos/openai/openai-node/releases?per_page=100' \ > .upgrade-releases.json jq -r --argjson from '[6,49,0]' --argjson to '[7,47,0]' \ -f .upgrade-between.jq .upgrade-releases.json \ > .upgrade-changelog.md grep -n -i 'breaking' .upgrade-changelog.md --paginate --slurp returns one array of pages, which is why the filter starts with .[][]. The test line skips prerelease tags such as -beta.1, whose parts aren't plain numbers. Each tag becomes an array like [7,44,0], and jq orders arrays lexically, element by element, so 7.10.0 lands above 7.9.0 where a string comparison would put it below. The output is newest first. That grep is your short list. For this range it catches at least the three entries from the top of this post: the Node 22 requirement and the two minor releases labelled breaking. The rest of .upgrade-changelog.md is what the agent reads when an error doesn't match any of them. !A long unrolled scroll of release notes under a magnifying glass, with one line glowing burnt orange among dozens of grey ones The breaking change you need is rarely in the major release's notes alone. Step 3: Bump in a throwaway worktree and save what broke You want the real failure list before the agent starts, and you don't want your main checkout sitting on a half-upgraded lockfile meanwhile. A detached git worktree is a second folder on the same commit, with no branch to clean up afterwards. git worktree add --detach ../app-probe cd ../app-probe npm ci npm install openai@7.47.0 npx tsc --noEmit --pretty false \ > ../app/.upgrade-tsc.txt 2>&1 npm test > ../app/.upgrade-tests.txt 2>&1 cd ../app git worktree remove --force ../app-probe npm ci comes first because a new worktree holds tracked files only, so there is no node_modules yet. --noEmit type-checks without writing any output. --pretty false matters when output goes to a file: pretty is on by default and adds colour and context, and the plain form is easier for you and the agent to read line by line. Both outputs are written back into the main folder, so they outlive the probe. --force is there only because the probe holds an uncommitted bump on purpose. Git's docs say remove refuses an unclean worktree without it. Keep that refusal on any worktree that holds work you care about. grep -c 'error TS' .upgrade-tsc.txt counts the type errors. If both files come back clean, the upgrade may be one line plus the Node change. Step 4: Write the brief and start the task Save this as .upgrade-brief.md, edited for your versions: Upgrade openai from 6.49.0 to 7.47.0. Read before editing: - .upgrade-changelog.md: every release note in range - .upgrade-tsc.txt, .upgrade-tests.txt: what broke 1. Run `npm install openai@7.47.0`, then `npx tsc --noEmit` and `npm test`. If the failures differ from the files, stop and say so. 2. Fix one failure per turn. Start each reply with the error line you fixed. 3. Name the release note each fix follows. If no note explains a failure, stop and ask. 4. Don't edit or delete tests. Don't add @ts-ignore, @ts-expect-error or `as any`. 5. 7.x needs Node.js 22. Update "engines" and the Node version in CI to match, in its own turn. 6. Done when tsc and npm test both pass. Rules 1 and 2 keep you and the agent on the same failure list, one entry per reply, which is what the review in step 5 needs. Rule 3 forces every change to point at a note, and rule 4 shuts the cheapest way to turn a suite green. Next, the recipe Hermes runs for every new task's worktree, from the recipe docs: # .hermes/worktree.toml copy = [".upgrade-*"] done_when = ["npx tsc --noEmit", "npm test"] copy brings files from your project folder into the same path in the worktree, but only files git ignores. If a pattern matches a tracked file, Hermes refuses the whole recipe and setup doesn't run. That is why the .gitignore line came first in step 2. done_when holds the default checks for the task's Done-When, here the same two commands as rule 6. The first time Hermes sees the recipe, and whenever it changes, it asks you to choose Run setup or Skip. Choose Run setup: per the docs, Skip "starts the session without setup." Commit the recipe and the .gitignore line together. Then press ⌘N (Ctrl+Shift+N on Windows and Linux), pick Claude Code or Codex, and type Read .upgrade-brief.md and follow it. The agent starts in a real terminal on a new hermes/ branch, in its own worktree, with that line as its first prompt. Your main checkout stays on 6.49.0 and keeps working while it runs. Hermes never answers a permission prompt for you, so when the agent asks to run npm install, you answer in its terminal. !The Hermes New task sheet with a task typed in and chips for Claude Code, the project, a new hermes/ worktree, Accept edits, model and effort, above a preview of the claude command The ⌘N New task sheet in Hermes 2.3.4: chips for Claude Code, the project, a new hermes/ worktree, Accept edits, model and effort, and the "Hermes will run" preview of the claude command. Step 5: Review one failure per turn, then land Because of rule 2, every agent reply should map to one line in .upgrade-tsc.txt or .upgrade-tests.txt. The "Agent replies" strip under the session shows one chip per reply with the lines it changed. Click a chip to see that reply's changes, or to restore your files to that point. ⌘G (Ctrl+Shift+G on Windows and Linux) opens the Review Desk, which shows changes by turn or by file, with risk flags. Read it by turn with the failure list open beside it. For each turn, check three things: the diff touches the code the error line points at, it follows the release note the agent named, and it leaves the tests alone. Switch to the by-file view for that last one. A line comment goes back to the agent that wrote the line. Say a turn silences the 7.44.0 error with a cast: comment on it and ask for the narrowing the release notes suggest, checking 'content' in event.item before reading message fields. !The Hermes Review Desk in By turn view, listing two turns with file and line counts, the first turn's diff of a TypeScript file, the changed-files list and a drafted commit message The Review Desk (⌘G) in By turn view in Hermes 2.3.4: two turns listed with their file and line counts, turn 1's diff, the changed-files list, and a commit message drafted from the turns. Without Hermes, the same review from inside the task's worktree is two commands. The agent's edits usually sit uncommitted until you land, so compare the working tree, not commits: git status --short git diff main -- '*.test.ts' The first lists every file the agent touched. The second should print nothing; any output there means a test was edited. When the checks pass, Land commits, opens a pull request or merges locally, into the branch the task started from. For a dependency upgrade, take the pull request: CI runs the Node versions and environments your laptop doesn't. Watch it with gh pr checks --watch. If CI disagrees, don't merge. Paste the failing job's output into a new .upgrade-ci.txt and tell the agent to read it. If you landed locally instead and main goes red, Land can be undone. What this doesn't replace Bots already open the bump PR, and both of the common ones put release notes in it. A Dependabot PR carries Release notes, Changelog and Commits sections. Renovate's docs say it filters the releases to the versions the PR updates and embeds them in the PR body. If you already use one, that is step 2 done for you. What they don't give the agent is your failure list and the rules. You don't need Hermes for the brief. Claude Code's overview lists updating dependencies among the jobs it handles, and the Codex CLI runs commands in your terminal too. Hand either one .upgrade-brief.md in a plain worktree and the method holds. Hermes adds the task's own branch and worktree, the per-turn review and Land with undo, all on your machine. It isn't affiliated with OpenAI, Anthropic or any agent vendor. I haven't run this upgrade on a real app myself. How many errors you get, and which notes they trace to, depends on which parts of the SDK your code touches. That is why you capture them before the agent starts. --- If this was useful I build Hermes IDE, free and source-available (BSL 1.1) on macOS, Windows and Linux: download it or star it on GitHub if the task-per-worktree flow saves you a broken main. If the app you're upgrading calls tools through an LLM SDK, AI That Acts is the book about that layer: tool calling, functions and a first working agent, in TypeScript. That is the code a release like 7.44.0 reaches into.  The whole AI in TypeScript series, from your first LLM call to agents in production, is here.

Gemini 4 Argon, 18 Days After GA: 50 on the Independent Index, 110M Tokens to Get There - Book: LLM Observability Pocket Guide - Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- Google says Gemini 4 Argon "leads the frontier on a majority of benchmarks". On the independent Artificial Analysis leaderboard, Argon (high) now scores 50. At the same high setting, Claude Opus 5.5 scores 52 and GPT-6 Astra scores 55. Argon went generally available on the paid Gemini API on October 13, as gemini-4-argon. That is 18 days of real use. On October 28, Google announced Gemini 4 Carbon, a stronger model almost nobody can call yet. What the independent numbers say Start with the leaderboard. Artificial Analysis ranks Argon (high) #13 of 58 on its Intelligence Index v5.0. It lists only the high setting for Argon, which is why the comparison above uses other models at high too. At max effort, Opus 5.5 scores 58 and Astra 57 on the same leaderboard, but Argon has no max entry to set against them. The score has moved since R&D World's October 1 check. R&D World reported 53 at the highest reasoning setting then. Artificial Analysis has since moved to Intelligence Index v5.0, which Google's Carbon post calls "recently updated," so the two numbers may not be directly comparable. I don't know whether that change explains the whole gap. Token use is the bigger story. Artificial Analysis says Argon generated 110M tokens to run the index, against a median of 70M, and calls that "somewhat verbose." At launch, R&D World put it at about 62,000 output tokens per index task versus about 27,000 for GPT-6 Astra. That verbosity is what makes the price promo matter. In the same R&D World piece, Argon cost $1.99 per index task at the introductory price and $3.98 at the standard price. Astra cost $3.26. On October 31 the leaderboard lists $7.68 per task for Argon (high), next to $3.46 for Opus 5.5 (high) and $3.24 for Astra (high). Those current numbers run on the v5.0 index, where Argon used 110M tokens, and Artificial Analysis prices Argon at the $4 / $20 per million tokens standard rate rather than today's promo. Both $3.98 and $7.68 use that list price, so the jump likely reflects the new index version; on its own it doesn't show the model got worse. !Editorial illustration of a glossy launch scorecard on a pedestal, while a long ribbon of tokens spills off a desk beside it and coils on the floor Launch charts measure the score. Your bill measures the tokens it took to get there. Google's own table already showed the coding gaps. Argon scores 55.0% on FrontierSWE v2 against Astra's 65.5%, and 57.4% on Terminal-Bench 4.0 against Opus 5.5's 66.4%, per R&D World's reading of Google's numbers. The same article reports that on Arena's agent leaderboard Argon ranked 8th overall when checked on October 1, from 3,417 sessions, with a rank range of 3rd to 16th. What developers report These are anecdotes from small groups, not studies. Read them as signals. Every's "vibe check" panel, with Dan Shipper and his team, was the most detailed early test, as summarised by BigGo. Argon was a strong structural editor. Shipper said it shaved days off an 8,000-word piece, and it generated more ideas than rival models in the panel's 3D and game tests. Where it fell down: it was weaker than Claude at checking and revising its own code. One racing game it built had a wall that made the track unplayable. Its writing tested about two grade levels harder to read than rivals'. Shipper called the Antigravity harness "a terrible harness." On Google's own developer forum, a thread titled "How bad is Gemini 4 Argon?" opened on October 16. The first post called its coding "underwhelming." A later reply described a large codebase where Argon finished a task, then kept rereading old specs and tests and started doubting work that was correct. That poster said it "prioritizes rules over correctness." Another reply granted that it is strong at knowledge work. A separate October 20 thread listed three problems: audio uploads not recognised in the Gemini web app, repeated approval prompts during pull request workflows in Antigravity, and quota that runs out fast during normal development. A reply the same day asked for reproduction steps, the Antigravity version and the OS. Not every test went against it. A Puter writing test of 140 runs, published October 19, found Argon answered opinion questions more directly than Gemini 3.8 Flash and hedged less, though it missed length limits more often. The forum reports and the Every panel point at the same weak spot: it doesn't check its own code well. Carbon: one outside number, the rest is Google's Gemini 4 Carbon is not something you can test. Google's announcement says it goes first to "a subset of cyber defenders," then to trusted partners through the Fairwind Program, and to developers and businesses after more work on guardrails, with no date. It was not in the Gemini API changelog as of October 31. One number does come from outside Google. Artificial Analysis lists Carbon (high) at 59, #1 on its index, and Google quotes that score in its post. The same page shows Carbon generated 200M tokens to run the index, against a 70M median, at $7.76 per task. So the verbosity pattern carries over. !Editorial illustration of a sealed glass display case holding a glowing orange gem, with a velvet rope in front and a small queue of silhouetted developers waiting outside Carbon on October 31: announced, benchmarked, and behind the rope for most developers. Everything else in the post is Google's own testing: SEC-bench Pro 88.2%, ExploitGym 54.8%, CWE-bench 71.0%, about half as many unsupported claims as Argon, 26% less overreach, and 0.0% attack success on Gray Swan's indirect prompt injection benchmark against Argon's 0.7%. Those may hold up. Nobody outside the access programs can check them yet. Treat them the way you would have treated Argon's launch table on day one. Measure it on your workload before December 1 Argon costs $2 / $10 per million input / output tokens through November 30, then $4 / $20 from December 1. It is not on the free tier. Output pricing includes thinking tokens, which is where a verbose model costs you. So measure tokens per task on your own prompts, and price them at both rates. With the Google Gen AI SDK for Python: from google import genai client = genai.Client() # reads GEMINI_API_KEY resp = client.models.generate_content( model="gemini-4-argon", contents="Review this diff for bugs: ...", ) u = resp.usage_metadata inp = u.prompt_token_count or 0 out = (u.candidates_token_count or 0) + ( u.thoughts_token_count or 0 ) now = inp * 2 / 1e6 + out * 10 / 1e6 dec = inp * 4 / 1e6 + out * 20 / 1e6 print(f"in={inp} out={out}") print(f"${now:.4f} now, ${dec:.4f} from Dec 1") Run your ten most common tasks through it, and the same tasks through the model you use today. Compare cost per finished task, not price per token. If you'd rather compare agents than raw API calls: I build Hermes IDE, so weigh this with that in mind. Hermes runs Antigravity CLI as a built-in agent next to Claude Code and Codex. Its model list comes from agy models, so Argon shows up if your Google account has it (Antigravity offers it on Google AI Pro, Ultra and Enterprise plans). Each task can get its own branch and worktree, so you can give Antigravity and Claude Code the same ticket on one repo. For the cost side, use the snippet above. Hermes is not affiliated with Google. !The Hermes New task sheet with its agent picker open, listing Claude Code and Codex as installed, then Antigravity CLI, Gemini CLI and other agents, plus Custom agent The ⌘N agent picker in Hermes 2.3.4, with Antigravity CLI among the supported agents. Have you moved real work to Argon since October 13? Did your token counts match the leaderboard's, or your own expectations? Tell me in the comments. --- If this was useful If you want to try Argon next to the agents you already use, Hermes IDE is free on macOS, Windows and Linux, and the source is on GitHub. Benchmark scores don't tell you what a model costs on your traffic. The LLM Observability Pocket Guide covers how to trace token use, cost and quality per request, and how to pick the tools that do it, so a model switch is a measured decision. 

Code generator開発記（Part 2） ! この記事は2023年7月3日に執筆されたものです。 こんにちは。Inflabのフロントエンドエンジニア、Lucasです。 前回の記事では、code generatorの必要性と、自前で実装するに至るまでの悩みを共有しました。今回は、code generatorをはじめ、schema validator、type guardをどのように実装したのかを簡単にご紹介したいと思います。 最初にお伝えしておくと、これはとてもシンプルなアイデアで、実装に複雑な技術は必要ありません。その前提で読み進めていただければと思います。 1. 目標と必要な機能 まず、前回の記事で立てた目標について改めて考えてみます。私はcode generatorを実装する前に、次のような目標を立てました。 ファイルまたはネットワークリクエストでOpenAPI specification JSONを直接パースし、望む形の結果をファイルとして生成できること。 フロントエンドエンジニアが、APIのschemaの変化にできるだけ手間なく対応できること。 フロントエンドエンジニアが、生成物として出力したいAPIを選択できること。 APIで宣言したresponseをJSON validatorで検証できるtype guardが自動生成されること。 バックエンドエンジニアが、フロントエンドの手間を気にせずAPIを作れること。 バックエンドエンジニアが宣言したSwaggerと実際のAPI responseの形が一致しているかを、integration testで検証できること。 特に最後の目標は、Elixirのdoc testの概念を借りたものです。昨年取り組んだ個人プロジェクトでテストを書いた際の感触が良く、GraphQLがサポートしているようなschemaの整合性検証も、近い形なら実現できるのではないかと期待していました。特に私たちのバックエンドチームはテストを書くことを非常に重視しているので、OpenAPI specificationと実際のAPIの結果をテストする手軽な方法を提供できるだろうと考えました。 以下、OpenAPI specification JSONはOASと略します。 2. 使用したライブラリ まず、実装に使ったライブラリを紹介します。おそらくほとんどの方がご存じだと思います。 1. 実装 inversify 機能が増えるほど、コードベースは必然的に大きく複雑になるだろうと考えました。また、複数のオプションをサポートする予定だったので、実行時に動く実装も変わりうると見ていました。 そのため機能間の依存関係が複雑になりそうだと考え、悩んだ末に依存性注入（DI）のためのIoCコンテナを探すことにしました。私はinversifyを選びましたが、もう少し軽量に使いたいのであれば、typediという選択肢もあります。 json-schema-to-typescript 事実上、code generatorの中核となるライブラリです。Code generatorのほぼすべての実装は、このライブラリから望む結果を得るためのオブジェクトを作る作業だと言っても過言ではありません。例えば、次のようなJSON schemaがある場合、 { "title": "Example Schema", "type": "object", "properties": { "firstName": { "type": "string" }, "lastName": { "type": "string" }, "age": { "description": "Age in years", "type": "integer", "minimum": 0 }, "hairColor": { "enum": ["black", "brown", "blue"], "type": "string" } }, "additionalProperties": false, "required": ["firstName", "lastName"] } 次のように型を生成します。 export interface ExampleSchema { firstName: string; lastName: string; /** - Age in years */ age?: number; hairColor?: 'black' | 'brown' | 'blue'; } lodash object-traversal js-yaml オプションをYAML形式で受け取るつもりでインストールしました。今思えば、普通にJavaScriptで受け取っても問題なかったと思います。 jsonschema Code generatorが受け付けられるオプションを検証するために選びました。オプション関連のコードがif ~ elseだらけになるのを避けたかったのです。 2. ビルド、デプロイ TypeScript tsup esbuildベースのビルドツールです。速くて設定が簡単なので選びました。 Release Please ライブラリではありませんが、このGitHub Actionのおかげで手軽にデプロイできたので紹介します。 コミットメッセージの中にconventional commit形式で書かれたものがあれば、pull requestの作成からデプロイまで自動で行ってくれます。特にchange log、release、git tagなども自動で管理してくれるので、開発者はコードだけに集中できます。 ちなみに、Inflabのフロントエンドチームでは、monorepoで構成されたパッケージはchangesetsで、単独のリポジトリで運用しているパッケージはRelease Pleaseで管理しています。 commitlint 上で説明したRelease Pleaseの動作の仕組み上、開発者のミスを防ぐために導入しました。コードの変更がコミットされる前にコミットメッセージを検査し、conventional commitの形式に従っていなければコミットできないようにします。 コミットメッセージの一貫性を保ち、デプロイのためにコードを変更したのに、コミットメッセージの誤りでパッケージがデプロイされないという問題を防ぐのが狙いです。 3. テスト Jest 3. 実装の詳細 Code generatorは、大きく3つの段階に分けて実行されます。1つ目は前処理の段階です。正しいオプションが渡されたかを検証し、OASのJSONファイルを読み込みます。続いて、ユーザーが望む前処理ロジック（Pre process）を実行して、読み込んだOAS JSONを加工します。 2つ目の段階では、Taskと呼ばれる作業単位が並列に実行されます。どのタスクを実行するかは、そのタスクのオプションが正しく入力されているかどうかによって決まります。 ReactQueryTaskでは、前処理済みのOAS JSONから生成物に使うオブジェクトを作り、適切な名前と対応付けます。Query、url params、response、request bodyなど、必要と思われる要素をすべて抽出し、型とhooksに変換します。 TypeGuardTaskでは、ReactQueryTaskで作られたインターフェースを検証するtype guard関数を作ります。 SchemaValidatorTaskは、OAS JSONで宣言されたすべてのschemaを検証できるstandaloneのAjvインスタンスを生成します。Ajvについては、後ほど改めて詳しく紹介します。 Taskの並列実行が終わったら、最後の段階として、指定された場所へファイルを書き出します。フローチャートで表すと、次のようになります。 1. オプションの入力を受け取る ユーザーから希望するオプションをYAML形式で受け取るように構成しました。そのオプションは次のとおりです。 loader: fetch | file [required] schema: string [required] output-dir: string [required] react-query: [optional] output: string [required] fetcher: string [required] version: 3 | 4 [optional] default: 3 validator: [optional] output: string [required] typeguard: [optional] output: string [required] preprocess: [optional] filterDeprecated: boolean [optional] default: false extractEnum: boolean [optional] default: false apiPaths: - path: string [required] exact: boolean [optional] tags: - string [required] 2. 前処理 1. オプションの検証 YAMLファイルの有無の確認、node argumentsの入力チェック、kebab-caseのオプションのcamelCaseへの変換など、I/Oと副作用はConfigProviderが担当するようにしました。 一方、react-query、preprocessなど各オプションのパース、検証、値の設定は、ConfigLoaderを実装したオブジェクトが担います。各オブジェクトがパースしたオプションは、ConfigProviderに渡されます。こうしたのは、オプションごとの変更の影響を互いに切り離すためです。 例えば、react queryを担当するクラスは次のように実装されています。 import Ajv from 'ajv'; import type ConfigLoader from './ConfigLoader'; import type { JSONSchemaType } from 'ajv'; import { isNil } from 'lodash'; import { InvalidConfigException } from '../utils/exception'; type ReactQueryConfigOptions = { output: string; fetcher: string; version?: number; removeApiEndpointPrefix?: boolean; }; type YamlReactQueryConfig = { reactQuery?: ReactQueryConfigOptions; }; /** - 1. react-queryオプションのjson validationロジック */ const schema: JSONSchemaType<YamlReactQueryConfig> = { type: 'object', properties: { reactQuery: { type: 'object', nullable: true, properties: { output: { type: 'string' }, fetcher: { type: 'string' }, version: { type: 'number', nullable: true, default: 4, minimum: 3, maximum: 4 }, removeApiEndpointPrefix: { type: 'boolean', nullable: true }, }, required: ['output', 'fetcher'], }, }, additionalProperties: true, }; const validate = new Ajv({ useDefaults: true }).compile(schema); export type ReactQueryConfig = ReactQueryConfigOptions | false; export default class ReactQueryConfigLoader implements ConfigLoader<ReactQueryConfig> { /** - 2. react-queryのオプションの検証 */ load(optionCandidate: unknown) { if (!validate(optionCandidate)) { throw new InvalidConfigException('ReactQueryConfigLoader: not a react query config'); } /** - 3. react-queryのオプションを返す - 戻り値の型はReactQueryConfigOptions | false */ if (isNil(optionCandidate.reactQuery)) { return false; } return optionCandidate.reactQuery; } } ReactQueryConfigLoaderでパースしたオプションは、ConfigProviderのパブリックメソッドとして提供されます。 export class ConfigProviderImpl implements ConfigProvider { private loader: Loader; private options: Options; constructor() { /** - 1. yamlファイルの読み込み */ const yamlCandidate = this.loadConfig(); const reactQueryConfigLoader = new ReactQueryConfigLoader(); /** - 2. react-queryのオプションのパースを依頼 */ const reactQueryConfig = reactQueryConfigLoader.load(yamlCandidate); this.options = { reactQuery: reactQueryConfig, }; } private loadConfig() { const flags = args.parse(process.argv); const yamlPath = resolve(process.cwd(), flags.config); if (typeof yamlPath !== 'string') { throw new InvalidPathException('ConfigProvider: failed to open yaml config file'); } const yamlConfig = camelizeKeys(yaml.load(readFileSync(resolve(__dirname, yamlPath), 'utf-8'))); return yamlConfig; } getOptions(): Options { return this.options; } /** - 3. オプションが必要なオブジェクトにオプションを提供 */ taskOption(): TaskOption { return pick(this.options, ['reactQuery']); } } この構造のおかげで、ConfigLoaderを実装するだけでConfigProviderから使えるようになります。つまり、オプションはいつでも柔軟に追加・変更できます。 2. OpenAPI specificationファイルを受け取る 当初の目標の中に、「ファイルまたはネットワークリクエストでOAS JSONを直接パースし、望む形の結果をファイルとして生成できること。」がありました。これを実現するために、Loaderを実装したFileLoaderとFetchLoaderを作り、inversifyのfactory関数を使って、オプションに応じて使うべきローダーを選択するようにしました。 // inversify.config.ts const container = new Container(); /** - 1. IoCコンテナにオブジェクトを登録 */ container.bind<Loader>(TYPES.Loader).to(FetchLoader).whenTargetNamed('fetch'); container.bind<Loader>(TYPES.Loader).to(FileLoader).whenTargetNamed('file'); /** - 2. ファクトリー関数の設定 */ const loaderFactory = container .bind<interfaces.Factory<Loader>>(TYPES.LoaderFactory) .toFactory<Loader, ['file' | 'fetch']>((context) => { return (named) => { /** - 3. どのオブジェクトをどのように読み込むかを明示 */ return context.container.getNamed<Loader>(TYPES.Loader, named); }; }); // loader.ts const selectedLoader = container.getNamed(TYPES.LoaderFactory, options.loader); このように実装すると、既存のコードを修正せずに別のLoaderを追加できるというメリットがあります。例えば、databaseからOAS JSONを読み込む必要がある場合は、次のように対応できます。 class DatabaseLoader implements Loader { // ... } container.bind<Loader>(TYPES.Loader).to(DatabaseLoader).whenTargetNamed('db'); const databaseLoader = container.getNamed(TYPES.LoaderFactory, 'db'); 3. Preprocess pipe Preprocess pipeは、読み込んだOAS JSONをさまざまな条件で加工する工程です。DeprecatedなAPIを除外したり、特定のAPIだけを選択したり、enumの抽出を選択したりといった処理を担当します。preprocessのオプションはいろいろ選べ、こちらもオプションに対応する前処理のロジックが実行時に決まります。PreprocessPipeクラスが、先ほど紹介したfactory関数でPreprocessを実装したクラスを選んで配列に入れ、OAS JSONを順番に渡しながらロジックを実行します。サンプルコードは次のとおりです。 import { flow } from 'lodash'; class PreprocessPipe { private readonly preprocessOptions: PreprocessConfig; private readonly initialPreprocess: Preprocess[]; constructor( @inject(TYPES.PreprocessFactory) private readonly preprocessFactory: (name: string) => Preprocess, ) {} exec(swaggerSpecJSON: SwaggerSpecJSON | undefined): SwaggerSpecJSON | undefined { /** - 1. オプションで選択されたpreprocessの実装だけを選んで配列に入れる */ const preprocess = this.scheduledPreprocess(); const preprocessorFunctions = preprocess.map((preprocess) => preprocess.exec.bind(preprocess)); /** - 2. preprocessの実装を順番に実行 */ const result = flow(preprocessorFunctions)(swaggerSpecJSON); return result; } private scheduledPreprocess() { if (!this.preprocessOptions) { return []; } /** - enabled = trueのオプションのオブジェクトだけを読み込む */ const selectedPreprocess = Object.entries(this.preprocessOptions) .filter(([, enabled]) => enabled) .map(([name]) => this.preprocessFactory(name)); return this.initialPreprocess.concat(selectedPreprocess); } } こちらも、オプションを追加・削除しても既存のロジックに影響しないように実装しています。オプションの読み込みと検証、OAS JSONファイルの読み込み、preprocessの実行という前処理の段階は、使う場所によって実行のされ方が変わりうるため、できるだけ柔軟に設計しました。 3. ReactQueryTask 上で説明したとおり、json-schema-to-typescriptというライブラリで望む生成物を得るためにオブジェクトを作る工程です。ある意味で最も重要な部分なので、責務を細かく分けることに気を配りました。基本的な流れは次のとおりです。 Generator役のオブジェクトは、Parser役のオブジェクトに必要な部分のパースを依頼します。Parserが結果を返すと、それぞれの役割に応じて生成物を文字列として組み立てます。 Object Generatingを担当するオブジェクトは、パースされたデータを使って適切な形のオブジェクトを作ります。インターフェースとして生成される名前を指定して、json-schema-to-typescriptに渡します。例えば次のとおりです。 import { compile } from 'json-schema-to-typescript'; const queryString = queryStringParser.generate(oasJson); /** - { - "title": "V1UserListQueryString", - "type": "object", - "properties": { - "firstName": { - "type": "string" - }, - "lastName": { - "type": "string" - } - } - } */ const result = await compile(queryString); /** - export interface V1UserListQueryString { - firstName?: string; - lastName?: string; - } */ react-queryで使うhooksを作るオブジェクトも、同じくパースされたデータから適切な形のhooksの文字列を作ります。Hooksの場合は、template literalを使えば生成結果を簡単に宣言できます。 // template.ts export default function useMutationTemplate(name: string, method: string) { const type = ${name}Response; const optionType = UseMutationOptions<${name}Response, unknown, ${name}Variables, TContext>; return `export function use${name}Mutation<TContext>({ url, options, requestOptions, }: { url: string; options?: ${optionType}; requestOptions?: RequestOptions; }) { const result = useMutation<${type}, unknown, ${name}Variables, TContext>( (variables) => fetcher.${method}(url, variables, requestOptions), options, ) return { ...result, data: result.data?.data }; }`; } // ReactMutationGenerator.ts const result = useMutationTemplate('V1UserList', 'POST'); このようにgenerate layerで作られた結果は、それぞれstring[]の形でReactQueryTaskに戻されます。ここまでの説明は長くてピンとこないかもしれませんが、次のダイアグラムを見ていただければ、思ったよりシンプルな構造であることがおわかりいただけると思います。 4. TypeGuardTask 1. Type guardとは何か TypeScriptにおけるtype guardとは、複数の型を取りうる変数について、型の推論範囲を絞り込む（narrowing）ための関数のことです。例えば次のとおりです。 // type guard const isString = (value: unknown): value is string => typeof value === 'string'; const numOrStringDivide = (value: string | number) => { if (isString(value)) { return parseInt(value, 10) / 2; // value -> string } return value / 2; // value -> number }; フロントエンドでtype guardが必要になるのは、上の例のようにprimitive typeを区別したい場合もありますが、API responseの違いを見分けて処理を分岐させなければならない場面が出てくるからです。特に、通常の関数だけでなくcomponentの中でも便利です。例えば、API responseの形に応じてレンダリングするcomponentを切り替えたい場合です。こうした場面で、Inflabのフロントエンドチームはtype guardを積極的に使っています。 const isStudent = (value: Student | Teacher): value is Student => { return isPlainObject(value) && 'grade' in value && typeof value.grade === 'string'; }; export function Profile() { // Student | Teacher const { data } = useProfileQuery(); return ( <div>{isStudent(data) ? <StudentProfile data={data} /> : <TeacherProfile data={data} />}</div> ); } 2. Type guardの落とし穴 もちろん、GraphQLを使えば__typenameというプロパティを使って簡単に区別できます。しかし、そうでない場合は一般的に、API responseの形を必要な分だけパースする方法で型を区別することになります。上の例で言えば、ある値がオブジェクトで、'grade'というstring型のプロパティを持っていれば、'Student'型とみなすことになります。 これはTypeScriptの構造的型システム（structural type system）に起因するもので、よくまとまったドキュメントが多いので、ここではこれ以上説明しません。 本題に戻ると、Inflabのフロントエンドチームではtype guardを積極的に使っていますが、メリットと同じくらいデメリットもはっきりしていました。それは、type guard自体が誤って宣言されている場合、ロジック上の誤りが表に出てこないということです。代表的なのが、API responseが変わったのにtype guardを変更し忘れたケースです。 上の'Student'型で言えば、API responseが変更されて'grade'というプロパティの型が変わったり、プロパティ自体がなくなったりするケースが考えられます。こうなると、コードレビューをしても見つけにくいだけでなく、コンパイル時には何のエラーも発生しないため、ランタイムで問題が発覚することが多いです。 3. Type guardを自動で作る こうした理由から、type guardもOAS JSONから自動で作られたらどうだろう？ というアイデアが浮かびました。同僚と議論した結果、jsonschemaを使ってOASのschemaを検証できる形にし、ファイルとして出力する機能を実装することに決めました。実装全体の流れは次のとおりです。 まず、OAS JSONをJSON schemaの形に変換し、JSON schemaがサポートしていないプロパティを取り除きます。 import { draft7 } from 'json-schema-migrate'; import { unset } from 'lodash'; function convertSwaggerSpecToJSONSchema(swaggerSpec: SwaggerSpecJSON) { const componentSwaggerSpecs = swaggerSpec.components.schemas; /** - 1. OAS -> json schemaの形に変換 */ draft7( toJsonSchema(swaggerSpec.components.schemas, { strictMode: true, }), ); /** - 2. json schemaでサポートされていないプロパティを削除 */ removeUnSupportProperty(componentSwaggerSpecs); return componentSwaggerSpecs; } function removeUnSupportProperty(componentSwaggerSpecs: Record<string, JSONSchema4>) { const notSupportedType = ['default', 'deprecated', 'nullable', 'tsEnumNames', 'example']; traverse(componentSwaggerSpecs, ({ parent, key }) => { key ??= ''; if (notSupportedType.includes(key)) { unset(parent, key); } }); } こうして作ったJSON schemaを、template関数を使って文字列にすれば完成です。 const defaultTypeGuardTemplate = (jsonSchemas: Record<string, JSONSchema4>) => { return [ importTemplate(Object.keys(jsonSchemas)).trim(), 'const validator = new Validator();', Object.entries(jsonSchemas) .map(([key, schema]) => typeGuardTemplate(key, schema)) .join('') .trim(), ].join('\n\n'); }; const importTemplate = (keys: string[]) => { return ` import { Validator } from 'jsonschema'; import type { ${keys.join(', ')} } from './schema'; `; }; const typeGuardTemplate = (type: string, jsonSchema: JSONSchema4) => { const stringifyJsonScheme = JSON.stringify(jsonSchema); return ` export const ${is${type}} = (value: unknown): value is ${type} => { return validator.validate(value, ${stringifyJsonScheme}).valid; } `; }; const schema = convertSwaggerSpecToJSONSchema(oasJson); const result = defaultTypeGuardTemplate(schema); まだ完璧に動作するわけではありませんが、問題なく使えるようドッグフーディングを続けています。 5. SchemaValidatorTask 上でもお伝えしたとおり、Elixirのdoc testの考え方を取り入れ、バックエンドエンジニアが自分の宣言したOASと実際のresponseが一致しているかを、integration testの段階で検証できる機能を提供しようとしました。ライブラリにはAjvを選びました。バックエンド側で別途ライブラリをインストールしなくても、standaloneモードで動作するからです。 SchemaValidatorTaskの仕組みは、前の2つのタスクに比べてとてもシンプルです。OASをJSON schemaに変換し、standaloneモードを適用した生成物を返せば完了です。 import toJsonSchema from '@openapi-contrib/openapi-schema-to-json-schema'; import Ajv from 'ajv'; import standaloneCode from 'ajv/dist/standalone'; const schema = toJsonSchema(oasJson); const ajv = new Ajv({ schemas, code: { source: true, optimize: true, }, }); const result = standaloneCode(ajv); 6. 生成物の出力 各タスクでは、受け取ったstring配列から一定の基準で重複を取り除きます。ここまで問題がなければ、オプションで指定されたパスに結果を出力します。 4. 生成物の形 次のようなOAS JSONを受け取ったと仮定すると、各タスクの実行結果は以下のとおりです。 OAS.json{ "openapi": "3.0.0", "paths": { "/api/v1/category": { "get": { "operationId": "CategoryB2cController_findAll", "summary": "すべてのカテゴリー一覧", "description": "すべてのカテゴリー一覧を取得する。", "parameters": [], "responses": { "200": { "description": "", "content": { "application/json": { "schema": { "properties": { "statusCode": { "type": "string" }, "message": { "type": "string" }, "data": { "$ref": "#/components/schemas/CategoryAllResponse" } } } } } } }, "tags": ["カテゴリーenum API"] } } }, "components": { "schemas": { "CodeName": { "type": "object", "properties": { "code": { "type": "string" }, "name": { "type": "string" } }, "required": ["code", "name"] }, "JobGroupCodeName": { "type": "object", "properties": { "code": { "type": "string" }, "name": { "type": "string" }, "jobs": { "type": "array", "items": { "$ref": "#/components/schemas/CodeName" } } }, "required": ["code", "name", "jobs"] }, "CategoryAllResponse": { "type": "object", "properties": { "jobGroup": { "type": "array", "items": { "$ref": "#/components/schemas/JobGroupCodeName" } }, "jobSeekerStatus": { "type": "array", "items": { "$ref": "#/components/schemas/CodeName" } } }, "required": ["jobSeekerStatus"] } } } } ReactQueryTaskimport { useQuery, useMutation } from '@tanstack/react-query'; import type { UseQueryOptions, UseMutationOptions, QueryKey } from '@tanstack/react-query'; import fetcher from './Fetcher'; import type { RequestOptions } from './Fetcher'; export type Maybe<T> = T | null; export type OneOf<T> = T extends unknown[] ? T[number] : T extends readonly unknown[] ? T[number] : T; export type Exact<T extends { [key: string]: unknown }> = { [K in keyof T]-?: Exclude<T[K], undefined>; }; export type Values<T extends Record<string, unknown>> = T extends Record<string, infer U> ? U : T extends { [key: string]: infer U } ? U : never; export const enum ResponseStatus { OK = 'OK', SERVER_ERROR = 'SERVER_ERROR', DUPLICATE = 'DUPLICATE', NOT_FOUND = 'NOT_FOUND', BAD_REQUEST = 'BAD_REQUEST', BAD_PARAMETER = 'BAD_PARAMETER', CONFLICT = 'CONFLICT', UNAUTHORIZED = 'UNAUTHORIZED', FORBIDDEN = 'FORBIDDEN', PAYLOAD_TOO_LARGE = 'PAYLOAD_TOO_LARGE', TOO_MANY_REQUESTS = 'TOO_MANY_REQUESTS', } export type Success<T = unknown> = { statusCode: ResponseStatus.OK; message: ''; data: T; }; export type Error = { statusCode: Omit<keyof typeof ResponseStatus, 'OK'>; message: string; data: Maybe<''>; }; /** - @typeParam T - Response data type */ export type ApiResponse<T = unknown> = Success<T>; export const API_ENDPOINT = { V1_CATEGORY: '/api/v1/category', } as const; export const fetchV1Category = async (url: string, requestOptions?: RequestOptions) => fetcher.get<V1CategoryResponse>(url, requestOptions); export function useV1CategoryQuery({ queryKey, url, options, requestOptions, }: { queryKey: QueryKey; url: string; options?: Omit< UseQueryOptions<V1CategoryResponse, unknown>, 'queryKey' | 'queryFn' | 'initialData' > & { initialData?: () => undefined; }; requestOptions?: RequestOptions; }) { const result = useQuery<V1CategoryResponse, unknown>( queryKey, () => fetchV1Category(url, requestOptions), options, ); return { ...result, data: result.data?.data }; } /** - created by ResponseSchemaGenerator */ export interface V1Category { data: Maybe<CategoryAllResponse>; } export interface CategoryAllResponse { jobGroup: Maybe<JobGroupCodeName[]>; jobSeekerStatus: CodeName[]; } export interface JobGroupCodeName { code: string; name: string; jobs: CodeName[]; } export interface CodeName { code: string; name: string; } export type V1CategoryResponse = ApiResponse<V1Category['data']>; TypeGuardTaskimport { Validator } from 'jsonschema'; import type { CodeName, JobGroupCodeName, CategoryAllResponse } from './schema'; const validator = new Validator(); export const isCodeName = (value: unknown): value is CodeName => { return validator.validate(value, { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' } }, required: ['code', 'name'], }).valid; }; export const isJobGroupCodeName = (value: unknown): value is JobGroupCodeName => { return validator.validate(value, { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' }, jobs: { type: 'array', items: { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' } }, required: ['code', 'name'], }, }, }, required: ['code', 'name', 'jobs'], }).valid; }; export const isCategoryAllResponse = (value: unknown): value is CategoryAllResponse => { return validator.validate(value, { type: 'object', properties: { jobGroup: { type: 'array', items: { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' }, jobs: { type: 'array', items: { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' } }, required: ['code', 'name'], }, }, }, required: ['code', 'name', 'jobs'], }, }, jobSeekerStatus: { type: 'array', items: { type: 'object', properties: { code: { type: 'string' }, name: { type: 'string' } }, required: ['code', 'name'], }, }, }, required: ['jobSeekerStatus'], }).valid; }; SchemaValidatorTaskの結果は1つの複雑なファイルになるため、ここでは紹介を割愛します。 こうして作られた生成物は、フロントエンドのコードで次のように使えます。 import { useV1CategoryQuery, API_ENDPOINT } from './schema'; import { isJobGroupCodeName } from './guard'; const retrieveJobGroupCodeNameJobs = (value: unknown) => { if (isJobGroupCodeName(value)) { return value.jobs; } return []; }; export function A() { const { data } = useV1CategoryQuery({ url: API_ENDPOINT.V1_CATEGORY, queryKey: [API_ENDPOINT.V1_CATEGORY], }); return <div>{data.jobSeekerStatus.map((status) => status.name)}</div>; } バックエンドのテストコードでは、schema validatorを次のように使えます。 import validator from './validator'; describe('categoryAllResponse', () => { it('categoryAllResponseのOASと実際の結果は一致しなければならない。', async () => { const response = await controller.categoryAllResponse(); const result = validator.categoryAllResponse(result); expect(result).toBe(true); }); }); 5. まとめ 最初にお伝えしたとおり、一見すると工程は複雑に見えますが、まったくそんなことはありません。むしろ実装中に悩んだのは、責務をどう適切に分担するかと、SonarQubeのduplicate line code smellを避けるためにコードの重複をどう抑えるかでした。そのため、コード全体を通して次のような選択をしました。 各オブジェクトの責務を細かく分離するよう努めました。 各オブジェクトの横断的関心事はServiceというlayerに分け、必要なオブジェクトが必要な分だけ参照するように実装しました。 実行時に動くロジックを切り替える必要がある場合は、Strategyパターンを積極的に使いました。 同じロジックで小さな部分だけを変える必要がある場合は、Template Methodパターンを使いましたが、継承が1段階を超えないように実装しました。 実装した構造を簡単に表すと、次のようになります。 実装の紹介は以上です。 次回の記事では、code generatorを導入してから働き方がどのように変わったのかをご紹介します。

🔥 @golang just became the most-used language on @googlecloud. Java is close behind (on raw SDK API requests). Fastest growing? Python, Rust, C++, JS, and C# adding users and SDK requests rapidly. Install the Go SDK (googleapis/google-cloud-go), along with Genkit Go, and ADK Go!

Build a RAG SaaS with pipeshub-ai, gridex & MagicUI | Open-Source Stack In this episode we build a private company knowledge assistant by integrating pipeshub-ai + gridex + magicui + google-api-python-client + umami. documents become a permission-aware search and answer workspace Repositories used in this architecture: - pipeshub-ai: pipeshub-ai/pipeshub-ai - gridex: gridex/gridex - magicui: magicuidesign/magicui - google-api-python-client: googleapis/google-api-python-client - umami: umami-software/umami This episode explains one integrated product architecture, including interfaces, data flow, deployment, security, and failure handling. Subscribe for daily open-source product builds: youtube.com #GitHub #OpenSource #SoftwareArchitecture #Programming

This is a real gem: googleapis/release-please Thanks @seuros!!

this is from 4 months old Github commit before 3.6 flash "gemini-4-flash-preview": "gemma4" here is the link: github.com/googleapis/python-genai/commit/ca97c58… t.co

Google's Abirami Sukumaran Wants Java Agents to Stop Talking Directly to Databases An AI agent that can answer questions about a database is easy to demo. The difficult moment comes when a user says, “Book the 8 AM one,” and the system must remember which trip was discussed, enforce the user's identity, and create a real transaction without letting the model invent or alter privileged fields. That is the problem behind a recent Java release from Google Cloud. On October 6, 2026, Staff Developer Advocate Abirami Sukumaran and co-authors Stenal Jolly and Anubhav Dhawan announced version 1.0 of the MCP Toolbox Java SDK. Its notable idea is not that an LLM can generate SQL. It is that the model should see a narrow set of named tools while a separate service owns the database queries, authentication, and sensitive parameters. Google's announcement describes this pattern through a fictional transit concierge. This is a breakdown of that public release and its design, not a report of hands-on testing. The code and examples below are adapted from the published documentation, and vendor claims about production readiness should be validated against your own requirements. The useful question is not “Can the model write SQL?” Giving a model direct database access can look like the shortest route from a chat box to useful answers. But it also makes a language model part of the query-construction and permission boundary. A natural-language request may be ambiguous, credentials may be broader than intended, and a seemingly harmless tool can have effects that are hard to reverse. Google's example separates the pieces. The developer defines tools in a Toolbox configuration, including parameterized SQL. The Java application connects to the Toolbox server through MCP. The agent can discover and invoke the published tools, but the server-side configuration controls what those tools actually do. In the transit example, one tool searches schedules, another looks up pet policies, and a third books a ticket. The announcement's sample uses relational schedule and booking data alongside policy text and vector search. This isn't a guarantee that an agent is safe. A badly designed tool can still expose too much or perform a dangerous action. The architectural move is more modest and more useful: make the model choose among developer-defined capabilities instead of granting it a general-purpose database credential and hoping its prompt remains careful. What Sukumaran's announcement emphasizes The release note is written by a developer advocate, but the strongest part is its focus on boundaries that tend to get overlooked in prototype code. Bound parameters stay outside the model's control. The SDK supports binding values on the server side and pruning those bound parameters from tool definitions exposed to the LLM. Google gives tenant_id as an example: the application can bind a tenant context without inviting the model to choose a different tenant. This helps reduce one class of confused-deputy and cross-tenant mistakes. It does not replace authorization checks in the data service, and the bound value still needs to come from a trusted identity path. Authentication is separated from tool logic. Version 1.0 adds CredentialsProvider and AuthMethods abstractions. The announcement says credentials are resolved asynchronously for each request, with Google Application Default Credentials support included. That gives an application a defined place to obtain identity rather than hard-coding a token into a prompt or tool argument. The actual IAM policy, service identity, token audience, and user-to-data authorization remain the application's responsibility. The client is asynchronous-first. The official Java SDK repository says the client is designed around Java 17 or later, HttpClient, and CompletableFuture. Tool calls can be composed asynchronously, or deliberately blocked on with .join() when a synchronous boundary is appropriate. The distinction matters in server code: a convenience .join() inside a request path can occupy a thread while waiting, so the calling framework and execution model should guide how results are awaited. Transport and operations get practical attention. The release adds an HTTP transport abstraction, session tracking, protocol version selection, custom headers, default tool-parameter values, and a runtime warning when credentials are about to cross plaintext HTTP. None of those features is a complete operational policy. They are places where an integration library can make safer configurations easier to express and certain mistakes more visible. The team describes 1.0 as a stable, backwards-compatible foundation. That is the project's release statement, not an independent security certification or proof that every feature is equally mature across environments. A model-facing tool should be smaller than a database API A useful design exercise is to describe the model's job without mentioning the database. “Find up to five available trips between two cities” is a bounded capability. “Run arbitrary SQL against the production database” is not. For example, the Toolbox announcement shows a query-schedules tool with origin, destination, and a limit default. Its SQL uses bound placeholders rather than interpolating model-provided text into a query. The same design principle applies outside SQL: make each tool's purpose explicit, constrain its inputs, and return only the fields needed for the next decision. An application could start by publishing a read-only schedule search. After testing its access controls and result handling, it might add a booking operation that requires an authenticated user and validates the selected trip again at execution time. The booking tool should not trust a model's statement that a user has permission, nor should a hidden prompt be treated as authorization. A more complete flow has distinct checks: 1. Authenticate the person or service making the request. 2. Expose only the tools appropriate for that identity and workflow. 3. Validate tool arguments against application rules, not just a JSON schema. 4. Bind trusted context such as tenant or user identity outside the model's editable arguments. 5. Re-check availability and authorization when committing a transaction. 6. Return a clear result and record an audit event that avoids logging secrets. MCP standardizes how clients and servers exchange tool information. It does not decide which user is allowed to book, whether a query is safe, how a transaction is rolled back, or what belongs in an audit record. Those remain system design decisions. Where the example is strong, and where teams should be cautious The strongest part of the fictional transit flow is its separation between a conversational interface and the actual data operations. The agent handles the dialogue; dedicated tools define the operations; the Toolbox server mediates access. The example also makes a real application concern visible: follow-up messages depend on conversation state, while transactions depend on current business state. Those are not the same thing. Remembering “the 8 AM trip” should not mean trusting stale availability from an earlier turn. There are also gaps between a launch example and a production review. The blog's scenario is explicitly fictional. It does not establish measured latency, failure rates, resistance to prompt injection, or the security of a deployment. Nor does showing a sample schema prove that a general-purpose database agent is safe. Treat the post as a design proposal and a quickstart, then inspect the code and test your own environment. The repository itself includes an important qualification: its README says feature support is still in progress and is not at parity with the Python, JavaScript/TypeScript, and Go SDKs. It lists Java 17+ support and calls GraalVM Native Image experimental. Those notes are in the current project documentation. That is a useful counterweight to the polished “production workloads” framing in the launch post. Before adopting it, verify the exact features your application needs, the current release, and compatibility with your runtime and framework. The Google Cloud example uses Spring Boot, LangChain4j, AlloyDB, and Cloud Run. Those are choices for that demonstration, not prerequisites of the SDK. The repository describes the Java client as framework-agnostic and mentions Spring Boot, Quarkus, Jakarta EE, and custom Java applications as possible consumers. That flexibility is valuable, but you still need to decide where the MCP server runs, how it authenticates, and whether a managed database product is appropriate for your workload. A small evaluation checklist Before connecting an agent to real data, try this review against a disposable environment: - Tool scope: Can each tool be described as one narrow operation? Remove generic query or arbitrary-command tools unless there is a compelling, separately governed need. - Identity: Is caller identity derived from a trusted authentication flow, rather than from model-generated text? - Bound context: Are tenant, account, and user identifiers injected or checked by trusted code? Can the model see or change them? - Read versus write: Are search and mutation separate capabilities? Does a write revalidate current state and authorization at commit time? - Transport: Are credentials sent only over HTTPS? Does the service identity have only the permissions required for the operation? - Failure behavior: What happens if the model times out, the Toolbox is unavailable, a database call is retried, or the client loses its session? - Observability: Can you trace a tool invocation to an authenticated request without recording raw credentials or unnecessary personal data? - SDK fit: Does the Java SDK currently support the protocol and runtime features you actually depend on? Check the README and changelog rather than assuming parity from the version number. The checklist is less glamorous than an autonomous concierge demo. It is also where an enterprise integration becomes a system that can be reviewed and operated. The takeaway Sukumaran's MCP Toolbox Java SDK 1.0 announcement is interesting because it puts a concrete Java interface around a larger architectural argument: an agent should receive capabilities, not a blank check. The SDK offers building blocks for tool discovery, invocation, authentication, bound parameters, and asynchronous Java clients. Those blocks can make a carefully scoped integration easier to implement. They cannot supply your authorization model or prove the resulting application safe. The durable lesson is to keep policy and transaction rules in ordinary, testable services, let the model operate through deliberately small tools, and treat each write as an application decision rather than a conversational suggestion. For more practical writing on Java, backend systems, and AI infrastructure, subscribe for future guides and comparisons. Have you connected an AI agent to a database yet? What boundary did you find hardest to get right? Sources - Google Cloud: MCP Toolbox Java SDK v1.0 announcement, October 6, 2026 - Google Cloud's MCP Toolbox Java SDK repository - MCP Toolbox documentation