grafana

What broke when I moved Grafana Faro to OpenTelemetry JS 3.0 TL;DR: I opened the three PRs that move Grafana Faro's web SDK onto OpenTelemetry JS SDK 3.0. The version bump was the easy part. What took the time was a deep import the new exports maps reject, and then the tests. Jest was quietly running the Node build, and a disabled XHR instrumentation turned out to still be making spans. All three PRs are open and waiting on review. !The Faro epic on GitHub, Migrate to OpenTelemetry JS SDK 3.x, with the plan split into three steps Why Faro had to move at all OpenTelemetry JS SDK 3.0, experimental 0.300 and API 1.10 came out on Oct 20. They only work with each other, so you can't take one and leave the rest. Faro was on 2.x, 0.223 and API 1.9. That sounds like something you can sit on for a while, and for the stable 2.x packages you can, they're supported until about Oct 2027. The problem showed up somewhere else. The contrib browser instrumentations (document load, user interaction, long task) had already moved to core ^3.0.0. Install one of those next to @grafana/faro-web-tracing and npm resolves @opentelemetry/api to 1.9.1, cause Faro asks for <1.10.0. Instrumentation 0.300 imports logs from the API, and 1.9.1 doesn't have it. So your app breaks on a package you didn't even touch. The Faro team wrote this up as an epic with the plan. I picked up the steps. Three PRs, stacked I could have done it in one PR. It would have been one big diff touching about 30 files, and nobody wants to review that. | PR | What it does | Ships in | | --- | --- | --- | | #2358 | Moves tracing setup to @opentelemetry/sdk-trace | any 2.x minor | | #2363 | Removes deep imports into OTel packages | any 2.x minor | | #2364 | The actual 3.0 / 0.300 / 1.10 bump | open decision | The first two change nothing for users and work on 2.x, so they can merge whenever. Only the last one raises what your build needs. Stacking them means the risky diff is small and the boring ones can go first. WebTracerProvider is gone SDK 3.x drops @opentelemetry/sdk-trace-web and @opentelemetry/sdk-trace-base. Everything Faro used from them is already in @opentelemetry/sdk-trace 2.12, which is why this step can land on 2.x. The one real change is that WebTracerProvider.register() used to do several things for you. With the plain TracerProvider you do them yourself, in order: - provider.register({ - propagator: new FaroPausablePropagator(/* ... */), - contextManager: options.contextManager, - }); + trace.setGlobalTracerProvider(provider); + propagation.setGlobalPropagator(new FaroPausablePropagator(/* ... */)); + + const contextManager = options.contextManager ?? new StackContextManager(); + contextManager.enable(); + context.setGlobalContextManager(contextManager); The order matters more than it looks. The context manager has to be enabled before it's registered, and a second Faro instance must not register again over the first. I wrote tests for each of those paths, cause that's exactly the kind of thing that works on the happy path and breaks for the one app that passes its own propagator. Side effect I didn't plan for: the tracing IIFE went from 90,474 to 87,299 bytes minified. Dropping a package you don't use is a nice way to lose 3 KB. The exports map says no This is the one that would have broken the upgrade if you just bumped versions. All the 0.300 packages declare an exports map in package.json. Faro had imports straight into build/src of @opentelemetry/otlp-transformer, for the OTLP payload types and for createExportTraceServiceRequest and JSON_ENCODER. With an exports map, any subpath that isn't listed fails to resolve. It doesn't matter if the file is sitting right there. !Node fails with ERR_PACKAGE_PATH_NOT_EXPORTED and tsc fails with TS2307 when importing from otlp-transformer 0.300's build/src Pointing the paths at the new dist/ layout doesn't help either. Unlisted is unlisted. So I moved the OTLP types (IResourceSpans, IKeyValue, ESpanKind and the rest) into @grafana/faro-core, and swapped the exporter over to the public JsonTraceSerializer: - import { JSON_ENCODER } from '@opentelemetry/otlp-transformer/build/src/common/utils'; - import { createExportTraceServiceRequest } from '@opentelemetry/otlp-transformer/build/src/trace/internal'; + import { JsonTraceSerializer } from '@opentelemetry/otlp-transformer'; - const traceEvent = createExportTraceServiceRequest(spans, JSON_ENCODER); + const traceEvent: TraceEvent = JSON.parse( + new TextDecoder().decode(JsonTraceSerializer.serializeRequest(spans)) + ); That's one extra encode and parse per batch. I don't love it, though it's the public API in both 0.223 and 0.300 and the payload that goes out is the same. A snapshot test of the trace payload and the derived faro.tracing.* event now pins that down. One detail I'd point out to anyone doing the same thing. I didn't copy ESpanKind as a TypeScript enum. TraceEvent is public through faro.api.pushTraces, and enums from two different declarations aren't assignable to each other. If Faro shipped its own enum, any app building payloads with otlp-transformer's enum would stop compiling. A const object with a numeric union type keeps both working. This step also took @opentelemetry/otlp-transformer out of core and the OTLP transport, they only used its types. Without that, every Faro user would pick up 0.300's engines.node >=22.15.0, not just the ones using tracing. The bump, and what it found The version change itself is a few lines in package.json. core, resources and sdk-trace to ^3.0.0, the experimental packages to ^0.300.0, the API to ^1.10.0. Typecheck passed without a single change. That was honestly a bit suspicious. Jest was running the Node build Then Jest. Under Jest 30 with jsdom, OTel's packages resolve to their Node builds, not the browser ones. Nothing fails. You just test the wrong code. This is a one-line test that prints SDK_INFO from @opentelemetry/core 3.0, run once without the resolver and once with it: !The same jsdom test reports telemetry.sdk.language nodejs without the resolver and webjs with it, and passes both times SDK_INFO reporting nodejs inside a browser test environment is the tell. The 3.x migration guide has a Jest resolver for this, and adding it flips it to webjs. If you have your own Jest setup with OTel 3.x and jsdom, check this before you trust your green tests. A disabled instrumentation that kept tracing Running the tests against the real XMLHttpRequestInstrumentation (the old ones mocked it) turned up a bug that wasn't about 3.0 at all. FaroXhrInstrumentation overrides _patchOpen, and the override didn't check _enabled like the parent does. disable() leaves open patched, so a disabled Faro XHR instrumentation kept creating spans. !The new XHR test fails on the #2363 tree with one span after disable(), and all three tests pass on #2364 It's a one-line check. It's also the kind of bug that lives for years cause the unit tests mock the exact thing that would catch it. Size and tests !yarn build output for the tracing IIFE: 74.82 kB and 22.19 kB gzipped on #2363, 76.68 kB and 22.55 kB on #2364 The tracing bundle grows from 74.8 kB to 76.7 kB (22.1 to 22.5 kB gzipped) against #2363. The web-sdk and core bundles stay the same size, and there's no Node platform code in the browser bundle. On the #2364 head the full suite passes, 1,218 unit tests across seven packages with one skipped. !yarn quality:test on #2364: every package passes, 204, 56, 82, 193, 646, 31 and 6 tests What I checked outside the repo Unit tests only tell you so much for a change like this, so I packed the packages and tried them in real apps: - A webpack 5.54 app and a Vite 7 app both build and run. Fetch and XHR spans get exported with telemetry.sdk.language set to webjs, and the faro.tracing.* events still go out. - The .d.ts files typecheck with both bundler and node16 module resolution. - On the other end, Alloy's faro.receiver uses the collector's ptrace.JSONUnmarshaler, so span attributes with object, array and bytes values still parse. I also ran my own otel-js-upgrade codemod as a dry run on the #2363 tree. It proposed the same version bumps and no code changes. Which is the right answer, and kind of the point of this post: the parts that took time are the ones no codemod can see. What's still open The 3.0 PR has no ! in the title on purpose. Whether this ships as Faro v3 or a 2.x minor is still an open call in the epic. Faro's own API doesn't change, but the floor for your build does: webpack 5.54+, Jest 29.4+ or Parcel with packageExports, and Node 22.15+ for installs. Those are old versions to still be on, though a minor release breaking someone's build is still a minor release breaking someone's build. There's one thing I left alone. sendFaroEvents turns attribute values into strings with String(), so array and object values come out as [object Object] in the derived events. Arrays already did that on 2.x, 3.x just lets objects through too. That's a small follow up if the maintainers want JSON there. !PR #2364 on GitHub, the OpenTelemetry JS SDK 3.0 upgrade, open and waiting on review After this, the next step is #2340, replacing Faro's fetch and XHR instrumentations with the ones from the OpenTelemetry Browser SIG. That changes trace data, so it's a bigger conversation. If you're on Faro today Until the upgrade ships, if you add contrib browser instrumentations next to Faro, pin the last versions on 2.x: npm i @opentelemetry/instrumentation-document-load@0.68.0 \ @opentelemetry/instrumentation-user-interaction@0.67.0 \ @opentelemetry/instrumentation-long-task@0.67.0 That keeps @opentelemetry/api on one version and your app out of the logs import error.

いつの間にこんな機能が cog#1224

Grafana MCP 能让 AI 查监控面板、读指标，Apache-2.0 开源。排查告警，先取面板摘要，再按需读具体字段。 官方提醒：完整面板 JSON 可能挤占大量上下文。还得有 Grafana 9+ 和访问凭据，装个 MCP 不能凭空监控服务器。 grafana/mcp-grafana

VictoriaLogs 与 Loki、Elasticsearch、OpenSearch、SigNoz、ClickStack 的日志写入性能对比 > This blog post is also available in English: [VictoriaLogs vs Loki, Elasticsearch, OpenSearch, SigNoz and ClickStack on structured log ingestion]( victoriametrics.com) 在 VictoriaMetrics 内部，我们会持续地对自己的产品进行性能测试。这能帮助我们保持竞争力，让每一个版本都变得更好。随着越来越多的团队开始使用 VictoriaLogs ，也有越来越多的人问我们：它适合哪些场景？切换过来能获得什么？因此，我们把基准测试作为一个[公开 GitHub 仓库]( VictoriaMetrics/logs-benchmark)发布出来，任何人都可以运行它来复现我们的结果，我们也会继续以开源的方式维护它。 这不是一篇劝你在迁移到 VictoriaLogs 的广告。相反，如果你在采集结构化日志（无论来自 Kubernetes 还是其他地方），本文应该能帮你选择合适的工具，并了解适合你的权衡取舍。 对不同系统进行比较的基准测试总是充满争议，因为每个系统擅长的场景各不相同。本文只测试一个场景：写入结构化日志，即每一行都带有命名字段的日志，Kubernetes 上的许多服务都是这样输出日志的。每个系统都运行在一台相同规格的机器上。查询性能、集群模式和对象存储会在后续文章中介绍。 仅凭写入速度无法决定该选哪个系统，所以本文也会深入它们的内部实现： - 为什么 Loki 写入速度最快，但一个简单的 Dashboard 查询却要花上几分钟？ - 为什么 SigNoz 和 ClickStack 落后于 VictoriaLogs ，尽管三者都以列式方式存储日志？ - 为什么 Elasticsearch 和 OpenSearch 需要那么多磁盘空间？ 这些问题的答案能帮助你规划硬件和成本，避免在日志量增长时遇到意外。 测试与环境 我们测试了截至 2026 年 10 月 6 个系统的最新版本，配置尽量接近默认值： - VictoriaLogs [v1.53.0]( github.com/VictoriaMetrics/VictoriaLogs/releases/…) - Loki [3.7.8]( github.com/grafana/loki/releases/tag/v3.7.8) - Elasticsearch [9.5.5]( github.com/elastic/elasticsearch/releases/tag/v9.…) - OpenSearch [3.9.0]( github.com/opensearch-project/OpenSearch/releases…) - SigNoz [v0.145.0]( github.com/SigNoz/signoz/releases/tag/v0.145.0) - ClickStack [2.39.1]( github.com/hyperdxio/hyperdx/releases/tag/%40hype…) 我们调整了少量配置，例如 Loki 的写入限制，以便每个系统都能在单节点上达到其设计上限。VictoriaLogs 不需要任何调整，因为它的设计目标就是开箱即用地提供最佳性能（[文档]( docs.victoriametrics.com)）。[仓库]( github.com/VictoriaMetrics/logs-benchmark/tree/ma…)中说明了每一项改动。 为了让测试保持简单，每个系统都以单节点模式运行在独立的 Kubernetes 节点上（ GCP 的 n2d-standard-8 ，8 个 vCPU 、32 GB 内存）。SigNoz 和 ClickStack 将日志存储在 ClickHouse 中，而 ClickHouse 本身无法直接接收日志采集器发送的日志。因此每个技术栈都会在 ClickHouse 所在的节点上运行自己的 Collector ，这些 Collector 基于 OpenTelemetry Collector 构建。  我们同时向每个系统发送相同的日志。写入速率从 10k 行/秒开始，每 10 分钟增加 10k ，最高到 180k 行/秒。一个系统的最大写入速率，是指它在一个完整的 10 分钟阶梯内平均能成功存储的最高速率。如果系统发生崩溃，我们只统计它首次崩溃之前的阶梯，因为崩溃本身就是它的极限。  我们统计的是每个系统实际存储的行数，而不是它接收的行数。例如，SigNoz 和 ClickStack 在 Collector 收到日志后就会立即返回响应，因此 Collector 一旦崩溃，就会丢失已经被接收的日志。Loki 是个例外：它没有低成本的方法统计已存储的行数，但它会在返回响应之前把每一行写入 WAL （ Write-Ahead Log ，预写日志），所以对于 Loki ，我们统计的是接收的行数。 写入速率只是故事的一半。为了展示它与查询速度之间的关系，我们以 35k 行/秒的速率向 VictoriaLogs 、Elasticsearch 和 Loki 写入相同的日志，持续 4 小时，然后在冷缓存状态下对几条常见查询进行计时。我们会在后续文章中深入讨论查询。 每个系统接收的负载 日志可以是任何东西，从纯文本到包含数百个字段的 Wide Events （宽事件）。本测试选取了中间情况：每行包含 30 到 40 个字段的结构化日志，这在 Kubernetes 中很典型。其中约 20 到 30 个字段由应用写入，日志采集器再额外添加约 10 个字段，例如命名空间、节点和标签。 设想一个有 125 个节点的 Kubernetes 集群，每个节点运行 40 个属于不同应用的 Pod 。每个 Pod 的日志构成一个日志流，因此这个集群共有 5,000 个日志流。这是一个中等规模的集群，与我们在 VictoriaLogs 用户中见到的许多集群类似。  我们想看看每个系统基于自身设计能走多远。因此，每个系统都以它偏好的格式接收日志。例如，Loki 通常从 Grafana Alloy 接收 protobuf 格式。我们没有向所有系统发送同一种格式（如 OTLP ）。如果那样做，我们测量的就不仅是各系统存储日志的能力，还包括它们转换该格式的效率。 在实际场景中，日志通常以 JSON 形式输出到 stdout 。因此日志采集器需要对其进行转换，而这同样会消耗 CPU 。如果有足够多的读者感兴趣，我们也可以用一种通用格式（如 JSON 或 OTLP ）重新运行测试。 测试结果 主写入测试我们运行了 3 次，下面的结果取 3 次运行的中位数。完整的数据表格见[仓库]( github.com/VictoriaMetrics/logs-benchmark/tree/ma…)。 对于 SigNoz 和 ClickStack ，CPU 和内存包含了 Collector 、ClickHouse 及其辅助服务。 本文中提到的少数额外测试（例如查询测试）是仓库之外的一次性运行。 最大写入速率（行/秒）  - Loki 写入最多，达到 139k 行/秒。 - VictoriaLogs 位居第二，为 120k 。 - Elasticsearch 、OpenSearch 和 ClickStack 在 28k 到 48k 之间。 - SigNoz 达到 10k 行/秒，并在 20k 时崩溃。 一行日志约为 930 字节的 JSON ，因此 120k 行/秒约等于 106 MiB/s ，即每天约 9 TiB 的原始日志。 每个数字的背后都伴随着在查询速度、成本或日志类型上的权衡。我们会在[为什么会是这样的结果](#为什么会是这样的结果)中进行解释。 CPU 使用量（核） 各写入速率下的平均 CPU 核数。每条曲线终止于该系统的最大写入速率。  - 在 10k 行/秒（所有系统都能承受的唯一速率）下，VictoriaLogs 和 Loki 使用 0.4 核，ClickStack 0.9 核，Elasticsearch 1.0 核，OpenSearch 1.5 核，SigNoz 4.2 核。 - 差距随速率增长而扩大。在 40k 行/秒时，VictoriaLogs 和 Loki 使用 1.6 核，ClickStack 4.0 核，Elasticsearch 5.2 核。 - 在 120k 行/秒时，Loki 使用 6.1 核，VictoriaLogs 使用 7.2 核。 在本次测试中，CPU 是大多数系统的瓶颈。 内存使用量（ GiB ） 各写入速率下的峰值内存，不含操作系统的 Page Cache 。每条曲线终止于该系统能跟上的最后一个速率。  - VictoriaLogs 在所有速率下内存占用都最少，在 40k 行/秒时比 Loki 少 14 倍。 - Elasticsearch 和 OpenSearch 无论速率如何都使用约 17 GiB ，因为它们在启动时会预留固定大小的堆内存。默认情况下，Elasticsearch 会将堆大小设为机器内存的一半（[源码]( github.com/elastic/elasticsearch/blob/v9.5.5/dist…)）。OpenSearch 不会自动设置堆大小，所以我们给它配置了相同的大小。 - ClickStack 在 10k 行/秒时使用 1.3 GiB ，在 40k 行/秒时使用 2.2 GiB 。 - SigNoz 在 10k 行/秒时使用 3.1 GiB ，这是它在图表中唯一的数据点。 内存决定了机器需要多大、成本多高，以及还剩多少内存可用于缓存和查询。 磁盘空间 下图展示了每个系统存储一百万行日志所需的磁盘空间。作为参考，一百万行日志以原始 JSON 形式约占 887 MiB 。SigNoz 标为 N/A ，因为它的 Collector 反复崩溃，存储的行数太少，且多次运行之间的数据波动太大，无法进行比较。  - VictoriaLogs 所需磁盘空间最少，其次是 ClickStack 和 Loki。 - Elasticsearch 所需空间几乎是 VictoriaLogs 的 5 倍，OpenSearch 则是 10 倍。 日志通常会保留数周，因此磁盘空间增长很快。例如，以 50k 行/秒的速率保存 30 天的日志，VictoriaLogs 约需 7 TiB ，Loki 约需 12 TiB ，Elasticsearch 约需 33 TiB 。 磁盘写入量 下表展示了每个系统每存储一行日志向磁盘写入的字节数。这个数值会大于最终的磁盘占用，因为每个系统都会不时重写数据，以保持数据紧凑并便于搜索。N/A 表示该系统无法跟上对应的速率。 | 系统 | 10k | 20k | 40k | 80k | 120k | 150k | | --- | --- | --- | --- | --- | --- | --- | | Loki (protobuf) | 1,108 | 1,234 | 1,411 | 1,251 | 1,210 | N/A | | VictoriaLogs | 323 | 284 | 226 | 223 | 163 | N/A | | Elasticsearch | 1,393 | 1,815 | 1,917 | N/A | N/A | N/A | | OpenSearch | 2,295 | 3,197 | N/A | N/A | N/A | N/A | | SigNoz | 1,420 | N/A | N/A | N/A | N/A | N/A | | ClickStack | 414 | 461 | 572 | N/A | N/A | N/A | VictoriaLogs 每行写入磁盘的数据量同样最少，而且在更高速率下写得更少，因为一次写入更多行时压缩效果更好。Loki 每行的写入量是它的 3 到 7 倍，具体取决于速率。 这一点很重要，因为每块磁盘每秒能写入的字节数都有上限，而更快的磁盘也更贵。在我们的测试中，使用的是 GCP 上 500 GiB 的 pd-balanced 磁盘，最高写入速度为 280 MiB/s ，所以没有任何系统在耗尽 CPU 之前触及这个上限。尽管如此，VictoriaLogs 能让你使用更便宜的磁盘；在自有硬件上，它对 SSD 的磨损也要慢得多。 为什么会是这样的结果 Loki Loki 的高写入速率让我们感到意外，但它的设计可以解释这一点。Loki 的设计目标是存储大量日志，之后只在较短的时间范围内读取少数几个日志流。因此它只为流标签（这里是节点和应用）建立索引，并原样存储每一行日志。相比之下，VictoriaLogs 会把每一行拆分为字段，并分别存储每个字段，从而让后续的查询更快。在 120k 行/秒时，这会比 Loki 多消耗约 18% 的 CPU 。 向 Loki 发送日志的方式也很重要。Grafana Alloy 和 Vector 默认发送 protobuf ，而 Fluent Bit 、Fluentd 和 Logstash 发送 JSON 。在另一次单独的测试中，使用 JSON 时 Loki 只达到约 99k 行/秒，低于 VictoriaLogs ，因为处理 JSON 比处理 protobuf 消耗的 CPU 多得多。所以如果你使用 Loki ，建议优先选择 Alloy 或 Vector 。 Loki 的设计有 3 个代价。第一个代价是内存。Loki 会把每个流最新的日志行保存在内存中，以便将它们一起压缩。这意味着它的内存会随着流的数量增长。集群越大，流就越多；任何集群在部署期间流也会增多，因为每个新 Pod 都会产生一个新的流。为了观察这会占用多少内存，我们还分别用 10,000 和 15,000 个流运行了测试。  在 15,000 个流时，Loki 在 60k 行/秒时因 OOM 被杀死。而 VictoriaLogs 在流数量增加到 3 倍的情况下，仅多用了 0.2 GiB 内存。更多的内存意味着更大的机器，也意味着可用于缓存查询数据的内存更少。 第二个代价是磁盘。Loki 把日志行当作纯文本进行压缩。它不像 VictoriaLogs 和 ClickHouse 那样利用行内的字段来获得更好的压缩效果，因此它所需的磁盘空间是 VictoriaLogs 的 1.8 倍。 第三个也是最重要的代价是查询时间。Loki 只能通过标签来缩小查询范围。除此之外的任何条件，比如一个单词或一个字段值，都需要对匹配流中的每一行进行解压和扫描。在相同的日志上，前文提到的查询耗时如下。 | 查询问题 | 时间范围 | VictoriaLogs （秒） | Loki （秒） | | --- | --- | --- | --- | | 统计某个应用的错误数 | 1 小时 | 0.06 | 3.5 | | 统计每个应用的错误数 | 1 小时 | 0.59 | 171 | | 包含某个常见词的最新 100 行 | 1 小时 | 2.5 | 32 | | 每个应用的 p99 延迟 | 1 小时 | 1.9 | 302 | | 前 10 个请求路径 | 1 小时 | 8.8 | 被拒绝，超过 500 个 Series | 这些都是排查故障、构建 Dashboard 和告警时的日常问题。除第一条外，Loki 在所有查询上都要花费半分钟甚至更久，或者直接拒绝执行。第一条查询只通过标签读取一个应用的数据，即便如此，它也比 VictoriaLogs 慢了近 60 倍。 Loki 正在增加更快搜索字段的方法，例如 Bloom Filter （布隆过滤器），但这些功能仍处于实验阶段，默认关闭，并且在单节点上不可用，所以我们没有启用它们。 这些缓慢的查询并不意味着 Loki 总体上是一个糟糕的日志方案。它的设计目标是把日志低成本地保存在对象存储中，并在需要执行大型查询时横向扩展到许多查询工作节点上。在我们的单台机器上，统计一小时内每个应用的错误数花了 171 秒，大约使用了 8 核中的一半。粗略估算，若要在约 2 秒内得到结果，同样的扫描大约需要 300 个核。所以，如果你想把大量日志低成本地存储在对象存储中，并且要么只读取其中一小部分，要么愿意为大量 CPU 核付费以实现快速搜索，那么 Loki 适合你。 如果这超出了你愿意承担的成本，而你仍然需要 Dashboard 和搜索在几秒内返回结果，请继续往下读。 Elasticsearch 和 OpenSearch OpenSearch 是 Elasticsearch 的一个 Fork ，所以本节主要介绍 Elasticsearch ，最后再说明 OpenSearch 的不同之处。 Loki 几乎不建索引，并在查询时为此付出代价。Elasticsearch 则恰恰相反。它以 3 种形式存储每一行日志： - 该行的原始 JSON。 - 倒排索引，一个将每个单词和值映射到包含它们的日志行的字典，因此搜索可以直接定位到匹配的行。 - 再以列的形式存储一遍字段（ doc values ），用于排序和聚合。  这一切都让写入变得昂贵。在 40k 行/秒时，Elasticsearch 使用了 5.2 核，而 VictoriaLogs 和 Loki 只用了 1.6 核；它在约 48k 行/秒时耗尽了 CPU 。瓶颈在于 CPU ，而不是磁盘。Elasticsearch 会在返回响应前把每个请求写入磁盘，但我们在另一次单独测试中关闭了这一行为，写入速率几乎没有变化。 我们的测试使用了 [LogsDB]( elastic.co)，这是 Elasticsearch 针对日志推出的更新、更紧凑的索引模式。但即便如此，Elasticsearch 使用的磁盘空间仍是 Loki 的 2.6 倍，几乎是 VictoriaLogs 的 5 倍。 作为回报，Elasticsearch 查找数据的速度比 Loki 快得多。在 Loki 需要几分钟的查询上，Elasticsearch 只需几秒甚至更短。下面是它与 VictoriaLogs 在几类常见查询上的对比。 | 查询问题 | 时间范围 | VictoriaLogs （秒） | Elasticsearch （秒） | | --- | --- | --- | --- | | 统计包含某个常见词的行数 | 1 小时 | 6.5 | 0.10 | | 统计每个应用的错误数 | 1 小时 | 0.59 | 0.46 | | 每个应用的 p99 延迟 | 1 小时 | 1.9 | 3.9 | | 前 10 个请求路径 | 1 小时 | 8.8 | 14.7 | | 统计包含某个双词短语的行数 | 1 小时 | 4.4 | 175 | Elasticsearch 查找单个词或值几乎是瞬间完成的，在简单计数（例如统计每个应用的错误数）上也更快。而一个短语，比如 "connection timeout"，则花了它大约 3 分钟。 为了节省磁盘，它针对日志的默认配置不会存储每个单词在行中的位置，因此它必须重新读取日志行，才能判断两个词是否相邻。VictoriaLogs 在这类查询上更快，在百分位数和 Top 值查询上也更快。 Elasticsearch 之所以流行，是因为它能快速搜索任意字段。人们对它的抱怨通常集中在两点： - 成本。 为每个字段建立索引会消耗大量 CPU 、内存和磁盘。 - 运维。 运维大型集群需要投入大量精力。 Uber 同时遇到了这两个问题。索引成本高昂，而且 Uber 在每个区域运行了 20 多个集群，以确保一个集群故障不会影响全局（[Uber]( uber.com)）。Cloudflare 则不得不对部分日志进行采样，因为全部存储需要数百 TB 的空间（[Cloudflare]( blog.cloudflare.com)）。两家公司都把部分最大规模的日志负载迁移到了 ClickHouse 。 OpenSearch 的写入量略高于 Elasticsearch 的一半（ 28k 对 48k 行/秒），磁盘占用约为后者的两倍。它与 Elasticsearch 在两项默认设置上有所不同： - 它没有类似 LogsDB 的模式，因此对每行原始 JSON 的压缩没那么紧凑（[OpenSearch 文档]( docs.opensearch.org)），也不会预先排序。 - 它没有内置的日志模板，因此大多数字符串字段会被索引两次：一次按分词，一次按精确值。 OpenSearch 与 Elasticsearch 共享相同的内核，因此通过更强的压缩、排序以及完整的日志模板，你很可能可以弥补大部分差距。我们刻意让它保持接近默认配置，以展示开箱即用时的表现。 Loki 以牺牲查询速度为代价换取快速写入。Elasticsearch 以牺牲写入速度和存储为代价换取快速搜索。那么 VictoriaLogs 又提供了什么呢？ VictoriaLogs 如今，许多公司都在输出结构化日志，每一行都带有命名字段，例如日志级别、HTTP 状态码或请求耗时。VictoriaLogs 利用了这类日志的两个特性，使写入和查询都能保持高速。 1. 同一个流中的日志看起来很相似。它们的字段几乎相同，而且许多字段几乎不变。每一行的命名空间都相同，日志级别也只有少数几个取值。VictoriaLogs 会把每个流的日志存放在一起，并把每个字段存储在独立的列中，这比存储整行日志的压缩效果好得多。 2. 许多查询只需要少数几个字段。例如，统计每个应用的错误数只需要应用和级别两个字段。VictoriaLogs 只读取查询所需的列。  这种存储方式本身就让 VictoriaLogs 在磁盘上非常紧凑，聚合也很快，所以它在写入时只需构建轻量的索引和少量元数据。这就是为什么它能以 120k 行/秒的速率写入，是 Elasticsearch 的 2.5 倍，同时磁盘占用少 4.7 倍。正如前文所述，只有 Loki 的写入速度更快。更多细节请参阅 [VictoriaLogs 如何存储你的日志]( victoriametrics.com)。 在查询方面，VictoriaLogs 在几秒内响应了所有查询，并且在每一条查询上都比 Loki 快。与 Elasticsearch 相比则互有胜负，我们会在关于查询的文章中详细分析。为了保持低成本的写入和存储，VictoriaLogs 不会构建完整的倒排索引。其代价是，搜索一个常见词（出现在大量行中的词）会比较慢。而查找一个罕见词（例如用户 ID ）依然很快，因为 Bloom Filter 可以让它跳过不可能包含该词的数据。 VictoriaLogs 使用的内存也少得多。在 40k 行/秒时，Loki 使用的内存约为它的 14 倍（ 11.4 GiB 对 0.8 GiB ）；在 120k 行/秒时约为 5 倍（ 12.8 GiB 对 2.5 GiB ）。这些空闲内存并不会被浪费。操作系统会用它来缓存最近写入或读取的数据，因此许多查询（尤其是针对近期日志的查询）可以直接从内存而不是磁盘读取数据。它也为大量并发查询留出了空间，我们计划在后续关于查询的文章中对此进行测试。 公平地说，本次基准测试使用的是结构化日志，这正是 VictoriaLogs 的优势所在。而针对这类日志的日常工作，很大一部分也是用于 Dashboard 、告警和分析的聚合，这正是列式存储最能发挥作用的地方。如果你的日志是非结构化的，例如老旧应用和网络设备输出的纯文本日志，VictoriaLogs 依然能很好地写入和搜索它们，但收益会比结构化日志小一些。 SigNoz 和 ClickStack SigNoz 和 ClickStack 比 Loki 更让我们意外。两者都把日志存储在 ClickHouse 中，而 ClickHouse 和 VictoriaLogs 一样是列式数据库，所以我们原本预期它们的表现会接近 VictoriaLogs 。但事实并非如此。ClickStack 达到 46k 行/秒，SigNoz 只有 10k ，而 VictoriaLogs 为 120k 。在磁盘空间方面，ClickStack 排名第二，但每百万行仍需要 82 MiB ，是 VictoriaLogs 的 1.5 倍。 较低的写入速率来自两个搭建在 ClickHouse 之上的方案，来自于它们的 Collector 。ClickHouse 是一个通用数据库，无法直接接收常见格式的日志，因此需要由一个 OpenTelemetry Collector 先把日志转换成行数据。这个 Collector 消耗的 CPU 比 ClickHouse 本身还多。在 SigNoz 中，10k 行/秒时 Collector 使用了 2.7 核，而 ClickHouse 为 1.5 核；在 ClickStack 中，达到最大速率时两者分别为 3.3 核和 2.2 核。  更大的磁盘占用可能源于两个技术栈存储字段的方式。它们把日志存储在一张固定列的 ClickHouse 表中，但 [OpenTelemetry 日志格式]( opentelemetry.io) 只有少数几个固定字段，外加可以容纳任意自定义数据的 Attributes 。固定字段（如 Timestamp 、SeverityText 和 TraceId ）拥有独立的列，压缩效果很好。而 Attributes 则被放入少数几个 Map 列中，许多字段的值混杂在一起，因此压缩效果较差，查询也更慢。 VictoriaLogs 受到了 ClickHouse 的启发，但它是专门为日志而构建的（[FAQ]( docs.victoriametrics.com)）。它与 ClickHouse 有两点不同，这很可能解释了写入速率和磁盘空间上的大部分差距： - 无需 Collector 。 VictoriaLogs 可以直接接收日志采集器已经在发送的各种格式，例如 OpenTelemetry 、Loki 、Elasticsearch 、Datadog 和 Splunk 。 - 无需 Schema 。 日志行中的每个字段都有自己独立的列，而不仅仅是少数几个固定字段。 在本次测试中，基于相同设计的 SigNoz 使用的 CPU 几乎是 ClickStack 的 5 倍，主要消耗在其 Collector 上。Collector 把这些 CPU 花在了额外的元数据上，这些元数据可以加速查询并为其 UI 提供支持，我们会在关于查询的文章中探讨这一权衡。此外，它默认不设内存上限，而我们保留了这一默认配置。因此当它处理不过来时，会把日志行一直堆积在内存中，直到被 Kubernetes OOM Kill 。其他用户也报告过类似的 Collector 崩溃问题（[GitHub issue]( signoz#6128)）。 公平地说，SigNoz 和 ClickStack 是完整的可观测性技术栈，它们把日志、指标和链路追踪汇集在一处，并把每条链路关联到对应的日志。本次测试只衡量了其中的一部分：日志写入。两个技术栈也都建议随着数据量增长增加更多的 Collector 。ClickHouse 的文档按约 3 核处理 60k 事件/秒来规划单个 Collector （[ClickHouse 文档]( clickhouse.com)），而 SigNoz 自己的基准测试在一台 16 核机器上，于 ClickHouse 旁运行了 3 个 Collector ，达到约 55k 行/秒（[SigNoz]( signoz.io)）。在我们的测试中，为了对所有系统公平，每个系统都只分配了一台 8 核机器，所以 Collector 不得不与 ClickHouse 共享资源。集群测试中各个组件可以独立扩展，能够展示它们从更大规模的部署中能获得多少收益，这也在我们的计划之中。 关键结论 每个系统都根据自身的优先级和架构做出了不同的权衡。 - Loki 写入最多，因为它建立的索引最少。代价是查询缓慢，除非你为更多 CPU 核付费来加速查询。 - Elasticsearch 和 OpenSearch 为每一行的每一个词建立索引。这让它们能够非常快速地查找单词和精确值。但代价是写入时 CPU 消耗高，并且需要大量存储空间。 - SigNoz 和 ClickStack 把日志存储在 ClickHouse 中。ClickHouse 并非专为日志而构建，因此两个技术栈都需要额外的组件，这会拖慢写入速度并增加成本。如果你希望把 OTLP 日志、指标和链路追踪集中在一处，它们会比较适合你。 - VictoriaLogs 受到 ClickHouse 的启发，但它是直接为日志而构建的。它充分利用现代结构化日志的特点，实现快速写入、秒级查询响应，并且在内存和磁盘上的成本最低。如果你的日志是纯文本，收益会小一些。 本文在单台机器、普通云盘上测试了结构化日志。在后续文章中，我们计划深入介绍查询、集群模式，以及在 VictoriaLogs [支持对象存储]( VictoriaLogs#1155)后的对象存储测试。 你可以使用我们的[基准测试仓库]( VictoriaMetrics/logs-benchmark)复现主要的写入测试结果，也可以通过[快速入门指南]( docs.victoriametrics.com)试用 VictoriaLogs 。如果你发现我们的测试环境有错误，或者知道如何更好地调优某个系统，欢迎在[仓库]( github.com/VictoriaMetrics/logs-benchmark/issues)中提交 issue ，或在 [VictoriaMetrics 的 Slack]( slack.victoriametrics.com) 上联系我们。

Grafana MCP server: A Model Context Protocol (MCP) server for Grafana (3.6k stars) What it does All text below is quoted directly from the project README. >MCP server for Grafana Features: - Search for dashboards: Find dashboards by title, folder UID, tag, or starred status - List dashboard versions: List saved versions of a dashboard as compact metadata (version number, author, timestamp, save message) - Get dashboard summary: Get a compact overview of a dashboard including title, panel count, panel types, variables, and metadata without the full JSON to minimize context window usage - Get dashboard property: Extract specific parts of a dashboard using JSONPath expressions (e.g., $.title, $.panels[*].title) to fetch only needed data and reduce context window consumption - Patch dashboard: Apply specific changes to a dashboard without requiring the full JSON, significantly reducing context window usage for targeted modifications - Get panel queries and datasource info: Get the title, query string, and datasource information (including UID and type, if available) from every panel in a dashboard - List and fetch datasource information: View all configured datasources and retrieve detailed information about each. >This provides access to your Grafana instance and the surrounding ecosystem. Repository grafana/mcp-grafana Repository stats |Metric|Value| |:-|:-| |Stars|3,620| |Forks|481| |Last release|v2.1.1 (2026-10-20)| |Languages|Go 92.2%, TypeScript 6.6%, Python 0.7%| |Contributors|150| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|@claude (224 commits), @cursoragent (13 commits)| |AI co-authored commits|Claude: 227, Cursor: 13, Copilot: 1| Main contributor |Field|Value| |:-|:-| |Name|Ben Sully| |GitHub|@sd2k| |Commits to this repo|375 (top contributor)| |Bio|Senior Software Engineer at @grafana, working on AI/ML| |Company / Location|Grafana Labs / Sheffield| |Followers|112| ────────────────────────────── 📌 How many great GitHub repos have you discovered and completely forgotten about? Don't let another useful tool disappear into your Reddit feed. Get the community's best repos, organized by category, plus key takeaways from real developer discussions, delivered to your inbox twice a week. 📬 Never lose track of great repos again. Subscribe free.

Dashboard platform that dazzles instantly Grafana lets you build fast, interactive dashboards that pull from multiple sources like Prometheus and Elasticsearch. 🍒 Built by the Grafana Labs team grafana/grafana #opensource #typescript

OpenTelemetry for Beginners: Logs, Metrics and Traces in One Setup Picture this. A customer says checkout feels slow. You open the logs and find thousands of lines. Which request was theirs? Which step was slow? Was it just them, or everyone? Logs alone can't answer all of that. You need three kinds of data about the same request, and a way to connect them. That's what OpenTelemetry gives you. It has a reputation for being complicated, but the core idea is small. In this post you'll send traces, metrics and logs from a small FastAPI app into Grafana. The backend is one Docker command. The boilerplate is handled by zero-code instrumentation. You write a few lines yourself for the interesting parts. > Heads up on Python logs: In OpenTelemetry Python, traces and metrics are marked Stable, but logs are still marked Development. Logging works fine for this walkthrough. Just expect the logs side to change more than the other two. What OpenTelemetry is (and what it isn't) OpenTelemetry, or OTel for short, is an open standard and a set of tools for producing telemetry. That's the traces, metrics and logs your software emits about itself. It's a Cloud Native Computing Foundation project, and it grew out of two older projects, OpenTracing and OpenCensus. The key word is _producing_. OpenTelemetry doesn't store or display anything. You still need a backend for that. The win is that you instrument once, and your code doesn't care which backend receives the data. Here are the pieces you'll meet: | Piece | What it does | In this post | | --- | --- | --- | | API and SDK | The code you call, and the engine behind it that batches and exports data | Installed with opentelemetry-distro | | Instrumentation | Libraries that create telemetry for you (web frameworks, HTTP clients, databases) | FastAPI and logging | | OTLP | The format and protocol the data travels in | gRPC on port 4317 | | Collector | A service that receives, processes and forwards telemetry | Built into the Grafana image | | Backend | Where the data is stored and viewed | Tempo, Prometheus, Loki and Grafana | The three signals in plain words | Signal | Plain-English meaning | Question it answers | Example | | --- | --- | --- | --- | | Traces | The story of one request as it moves through your code | Where did this request spend its time? | POST /orders took 380 ms, and 340 of that was charge_card | | Metrics | Numbers measured over time | How often? How slow? Is it getting worse? | Declined orders per minute | | Logs | Timestamped messages, in words | What exactly happened? | Payment declined: keyboard | Two terms you'll see everywhere. A span is one step inside a trace, like "charge the card". A trace is a tree of spans. Spans carry attributes, which are key-value details such as order.item = keyboard. Which signal do you reach for first? It depends on the question: !Mermaid Diagram The real power is in the links. A metric spike leads you to a trace. A span leads you to the logs written during it. That last jump works because each log record carries the trace ID and span ID of the request that wrote it. OpenTelemetry adds them for you. What we'll build A tiny orders API with one endpoint, POST /orders. It "charges a card", which is just a short sleep with a 25% chance of being declined. That gives you slow requests and failures to look at. Here's how the data flows: !Mermaid Diagram Each request will produce a trace shaped like this: !Mermaid Diagram The official docs make a useful point here. Automatic instrumentation captures what happens at the edges of your system, like inbound and outbound HTTP requests. It can't see what's going on inside your own code. So the top span is free. The two inner spans, the counter and the histogram are yours. You'll need Python 3.10 or newer (that's what OpenTelemetry Python supports), Docker, and a terminal. Step 1: Start the backend docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm This is Grafana's otel-lgtm image. It bundles an OpenTelemetry Collector, Prometheus for metrics, Tempo for traces, Loki for logs and Grafana to look at all three. The ports: - 3000 is the Grafana UI - 4317 is where OTLP over gRPC arrives - 4318 is where OTLP over HTTP arrives Open localhost and log in with user admin and password admin. > Development only: Grafana says this image is intended for development, demo and testing. The data lives inside the container, so it's gone when the container is. Don't use it as your production backend. Step 2: Set up the project mkdir orders-api && cd orders-api python3 -m venv venv source venv/bin/activate pip install fastapi "uvicorn[standard]" opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-logging opentelemetry-bootstrap -a install On Windows PowerShell, activate with .\venv\Scripts\Activate.ps1 and use python instead of python3. What each piece does: - opentelemetry-distro brings the API and SDK, plus two commands: opentelemetry-bootstrap and opentelemetry-instrument. - opentelemetry-exporter-otlp lets the app send data over OTLP. - opentelemetry-instrumentation-logging connects Python's logging module to OpenTelemetry. - opentelemetry-bootstrap -a install looks at what's installed and adds the matching instrumentation. Here, that means FastAPI. Run bootstrap _after_ you install FastAPI, because it decides based on what it finds. Step 3: Write the app Create main.py: import asyncio import logging import random import time from fastapi import FastAPI, HTTPException from opentelemetry import metrics, trace from pydantic import BaseModel, Field app = FastAPI(title="orders-api") # Let INFO records through. logger = logging.getLogger("orders") logger.setLevel(logging.INFO) tracer = trace.get_tracer("orders.tracer") meter = metrics.get_meter("orders.meter") orders_processed = meter.create_counter( "orders.processed", description="Orders processed, by outcome", ) payment_duration = meter.create_histogram( "payment.duration", unit="ms", description="Time spent charging the card", ) class Order(BaseModel): item: str quantity: int = Field(ge=1, le=10) class PaymentDeclined(Exception): """Raised when the pretend payment provider says no.""" async def charge_card(amount: float) -> None: started = time.perf_counter() with tracer.start_as_current_span("charge_card") as span: span.set_attribute("payment.amount", amount) try: await asyncio.sleep(random.uniform(0.05, 0.4)) # pretend to call a payment API if random.random() < 0.25: raise PaymentDeclined("card declined") finally: payment_duration.record((time.perf_counter() - started) * 1000) @app.post("/orders") async def create_order(order: Order): with tracer.start_as_current_span("process_order") as span: span.set_attribute("order.item", order.item) span.set_attribute("order.quantity", order.quantity) logger.info("Processing order: %s x%d", order.item, order.quantity) try: await charge_card(round(order.quantity * 9.99, 2)) outcome = "paid" logger.info("Order paid: %s", order.item) except PaymentDeclined: outcome = "declined" logger.warning("Payment declined: %s", order.item) span.set_attribute("order.outcome", outcome) orders_processed.add(1, {"outcome": outcome}) if outcome == "declined": raise HTTPException(status_code=402, detail="Payment declined") return {"status": "paid", "item": order.item} You don't need to memorise any of it. These are the parts worth understanding. Spans tracer.start_as_current_span(...) opens a span for the length of the with block. A span opened inside another one becomes its child. That's how charge_card ends up under process_order. set_attribute attaches details. Put anything here that will help you debug later, like the item or the amount. There's a nice default too. If an exception escapes a with span block, the SDK records it on the span and marks the span as an error. That's why a declined payment will show up as a red charge_card span, with no extra code from you. Metrics Two instruments, two jobs: - Counter: counts things, and only goes up. We add 1 for every order, labelled with its outcome. - Histogram: records measurements like durations. It lets you ask for percentiles later, such as "how slow are the slowest 5%?" I record the duration in milliseconds on purpose. The default histogram buckets are tuned for millisecond-sized values: 5, 10, 25, 50 and so on up to 10,000. Record seconds instead and almost everything lands in the first bucket, so your percentiles turn to mush. > Careful with metric attributes: Every unique combination of attribute values creates a new time series. outcome has two values, which is fine. A user ID or order ID would create millions. Keep high-detail values like order.item on spans, where they're cheap. Logs There's no special OpenTelemetry logging call here. You keep using Python's logging, and OpenTelemetry attaches a handler that ships the records. One gotcha: Python only lets WARNING and above through by default. So we set our logger to INFO, or the "Processing order" lines would silently vanish. > Following an older tutorial? Before OpenTelemetry Python 1.40.0 you had to set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to turn on log instrumentation. In current versions you don't, as long as opentelemetry-instrumentation-logging is installed. Step 4: Run it with auto-instrumentation Set a few environment variables, then start the app through opentelemetry-instrument. macOS and Linux: export OTEL_SERVICE_NAME=orders-api export OTEL_TRACES_EXPORTER=otlp export OTEL_METRICS_EXPORTER=otlp export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 export OTEL_METRIC_EXPORT_INTERVAL=5000 opentelemetry-instrument uvicorn main:app --port 8000 Windows (PowerShell): $env:OTEL_SERVICE_NAME = "orders-api" $env:OTEL_TRACES_EXPORTER = "otlp" $env:OTEL_METRICS_EXPORTER = "otlp" $env:OTEL_LOGS_EXPORTER = "otlp" $env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317" $env:OTEL_METRIC_EXPORT_INTERVAL = "5000" opentelemetry-instrument uvicorn main:app --port 8000 What the variables do: | Variable | What it does | | --- | --- | | OTEL_SERVICE_NAME | The name shown in Grafana. Skip it and your data shows up as unknown_service. | | OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER | Where each signal goes. otlp means send it over OTLP. | | OTEL_EXPORTER_OTLP_ENDPOINT | Where the backend listens. This is the gRPC default, shown so you know where to change it. | | OTEL_METRIC_EXPORT_INTERVAL | Milliseconds between metric exports. Set low here so metrics show up quickly (more on that below). | Your code contains no SDK setup, no exporter wiring, no FastAPI hooks and no log handler. opentelemetry-instrument did all of that before your app started. It also creates the POST /orders span and some HTTP metrics for every request. > Skip --reload and --workers for now. The auto-reloader spawns child processes and can break instrumentation. And with uvicorn --workers, the official troubleshooting guide says you get traces and metrics but no logs. For several workers, it recommends Gunicorn with Uvicorn workers, which exports all three. Step 5: Send some traffic In a second terminal, create load.py: import json import random import urllib.error import urllib.request URL = "http://localhost:8000/orders" for _ in range(40): body = json.dumps({ "item": random.choice(["keyboard", "mouse", "monitor"]), "quantity": random.randint(1, 3), }).encode() request = urllib.request.Request(URL, data=body, headers={"Content-Type": "application/json"}) try: urllib.request.urlopen(request) print("paid") except urllib.error.HTTPError as error: print("declined" if error.code == 402 else f"error {error.code}") Run it a couple of times: python load.py A stream of paid and declined scrolls past. Roughly one in four orders is declined. Step 6: Explore it in Grafana Open Grafana and use Explore from the left menu. Pick a data source at the top, one signal at a time. Menu names move around between Grafana versions, so don't worry if yours looks slightly different. Traces (Tempo) Choose the Tempo data source and search for service orders-api. Open any trace. A waterfall appears: POST /orders at the top, process_order under it, charge_card under that. Each bar's length is how long that step took. Now find a declined one. Its charge_card span is marked as an error, with the exception attached. Metrics (Prometheus) Choose Prometheus and try these queries: sum by (outcome) (rate(orders_processed_total[1m])) histogram_quantile(0.95, sum by (le) (rate(payment_duration_milliseconds_bucket[1m]))) The first shows paid versus declined orders per second. The second shows the 95th percentile payment time. Prometheus's default OTLP translation adjusts metric names: dots become underscores, counters gain _total, and the unit becomes a suffix (ms turns into milliseconds). If autocomplete shows a slightly different name, your setup uses another translation setting, so use what it shows. Logs (Loki) Choose Loki and run: {service_name="orders-api"} Your "Processing order" and "Payment declined" lines show up. Click one to expand it, and look for its trace ID. Follow one failure across all three This is the payoff. Try it: 1. In Prometheus, spot the declined rate. 2. In Tempo, open a trace with a red charge_card span. 3. Jump from that span to its logs, and read exactly what happened. The image's Grafana comes pre-wired so you can click from a trace to its logs, and from a log line back to its trace. If a link is missing in your version, query Loki directly. With Loki's native OTLP ingestion, the service name is an index label and the trace ID is structured metadata. So you can filter on the ID you copied from Tempo: {service_name="orders-api"} | trace_id="PASTE_THE_TRACE_ID_HERE" That's one request, seen through all three signals. Why your metrics showed up later than your traces Each signal is batched and sent on its own timer: | Signal | Default delay between exports | Variable | | --- | --- | --- | | Logs | 1 second | OTEL_BLRP_SCHEDULE_DELAY | | Traces | 5 seconds | OTEL_BSP_SCHEDULE_DELAY | | Metrics | 60 seconds | OTEL_METRIC_EXPORT_INTERVAL | Without the 5-second setting we used, you'd wait up to a minute to see your first metric. It's a classic first-day surprise. Do you need your own Collector? In this setup, the Collector inside the Grafana image does the job. You'll want your own as you grow. The OpenTelemetry docs give a few reasons. It gives many services one shared place to send data. It can gather traces across services running on different hosts. And it's a central spot to process data before it reaches a backend. It also keeps backend details out of your apps, so you can switch backends without touching your code. Their advice is that unless you have a single service or you're just experimenting, you'll want a Collector in production. A handy debugging trick When data isn't arriving, run a bare Collector that just prints what it receives. Save this as otel-collector-config.yaml: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 exporters: debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] exporters: [debug] metrics: receivers: [otlp] exporters: [debug] logs: receivers: [otlp] exporters: [debug] Stop the Grafana container first, since both want the same ports. Then run: docker run -p 4317:4317 -p 4318:4318 \ -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \ otel/opentelemetry-collector:latest \ --config=/etc/otel-collector-config.yaml On Windows, use the full path to the file instead of $(pwd). Restart your app and send traffic. Every span, metric and log shows up in the Collector's output. If you see them there, your app is fine and the problem is further downstream. Common first-day problems Most of these are explained above, so this is just an index: | Symptom | Where to look | | --- | --- | | No data anywhere, and no errors | The app wasn't started through opentelemetry-instrument, or the endpoint is wrong. Run the debug Collector above to check. | | Metrics are missing at first | The export delay table above | | INFO logs are missing | The logging gotcha in Step 3 | | Logs vanish with --workers, or everything vanishes with --reload | The callout in Step 4 | | Everything disappeared | The note in Step 1 about where the image keeps its data | From demo to production The demo is deliberately simple. A real setup adds a few things: - A real backend. Pick a managed service or a properly run self-hosted stack in place of the demo image. - Sampling. You usually don't keep every trace. Setting OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 keeps about 10%. - Care with sensitive data. Don't put passwords, tokens or personal data into attributes or log messages. The OpenTelemetry docs have a page on handling sensitive data. - Consistent names. OpenTelemetry has semantic conventions, a shared vocabulary for attribute names like http.route. Following them keeps your data readable across tools. And when one service calls another over HTTP, the instrumentation passes the trace context along in headers. Both services then show up in one trace. That's context propagation, and it's the feature that makes tracing really shine across a system. Where to go next - The OpenTelemetry Demo. A multi-service shop with telemetry already wired in. It's the best place to see tracing across services. - Manual instrumentation for Python. More on spans, events, links and metric instruments. - Collector processors. Filter, transform and sample data before it leaves. - More libraries. Add httpx, SQLAlchemy or a database driver, then rerun opentelemetry-bootstrap -a install. It picks up new libraries on its own. Recap - OpenTelemetry produces telemetry. A backend stores and shows it. - Traces show one request's journey, metrics show trends, logs tell the story. Their power comes from being linked. - opentelemetry-instrument gives you request spans and log shipping for free. You add spans and metrics for your own business logic. - Set OTEL_SERVICE_NAME, and shorten the metric interval while you're learning. - Keep metric attributes low-cardinality. Put the detail on spans. - The otel-lgtm image is a great sandbox, not a production backend. Instrument, export, then follow one request across all three signals. Everything else builds on that. - * * _Details checked against the OpenTelemetry documentation and the grafana/docker-otel-lgtm project on October 25, 2026. Tooling moves quickly, so if a command or menu behaves differently on your version, the official docs win._ Sources - OpenTelemetry Python: Getting Started by Example - OpenTelemetry Python: status and releases - Python logs auto-instrumentation example - Troubleshooting Python automatic instrumentation - OpenTelemetry SDK environment variables - grafana/docker-otel-lgtm on GitHub - Grafana: An OpenTelemetry backend in a Docker image --- Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.

cdk8s vs Tanka vs Kapitan for Kubernetes config Pick cdk8s if your team writes TypeScript and wants the compiler to reject a bad value before anything renders. Pick Tanka if you want a small, fast, Kubernetes-only toolchain and accept Jsonnet as the language. Pick Kapitan if one inventory has to drive Kubernetes and Terraform and you would rather keep values in YAML than maintain template code. All three rendered the same app to identical manifests and passed the derived-value test that Helm and Kustomize failed in the companion post. The example builds the app with each tool, and ./check exits 0 only when the normalized manifests match per environment. Versions, commands and raw output behind every number are in its NOTES.md. The app and the check The spec is the companion's: namespace shop, apps web and api with a Deployment, Service and ConfigMap each, restricted Pod Security, dev and prod, PodDisruptionBudgets in prod only, and an api ConfigMap entry web_url built from the web Service port. - Tanka 0.39.4, jsonnet-bundler 0.6.0, k8s-libsonnet 1.36. - cdk8s 2.70.111 in TypeScript 5.9.3, using the generated low-level Kube* classes for Kubernetes 1.36. - Kapitan 0.36.3 with the kadet-hub kubernetes generator 1.2.0 pinned by digest, as in the companion minus endpoints.json. check sorts objects and keys and drops one label, app.kubernetes.io/managed-by, which only the Kapitan generator sets and which names the tool. Nothing else is ignored. Changing one replica count in a copy makes it exit 1. $ ./check dev: tanka = kapitan (6 objects) dev: cdk8s = kapitan (6 objects) prod: tanka = kapitan (8 objects) prod: cdk8s = kapitan (8 objects) How much you write Hand-written files only; lock files, fetched libraries and generated code are listed separately. | | web only | both apps | fetched or generated | |---|---|---|---| | Tanka | 8 files, 138 lines | 8 files, 167 lines | k8s-libsonnet, 30037 lines | | cdk8s | 5 files, 148 lines | 5 files, 159 lines | imports/k8s.ts, 41214 lines, committed | | Kapitan | 5 files, 65 lines | 6 files, 101 lines | generator, 4166 lines Python | For the Helm vs Kustomize vs Jsonnet comparison: Helm needed 156 lines and Kustomize 299 for both apps in the companion. The gap is about who owns the template. Tanka and cdk8s ship building blocks, not an opinion on what a Deployment looks like, so the example carries its own: lib/shop/app.libsonnet (58 lines) and app.ts (87 lines). Both spell out fields Kubernetes would default, such as probe thresholds and defaultMode: 420, to match the generator. In Kapitan the template is the generator, maintained elsewhere and pinned by digest. cdk8s has an opinionated layer too, cdk8s+. An 11-line cdk8s-plus-36 program for the web Deployment produced a cdk8s.io/metadata.addr selector, 1000m/512Mi requests, 1500m/2048Mi limits, a read-only root filesystem and a TCP startup probe. Reasonable defaults, but not the spec's, so the example stays on the low-level classes. Adding the second app | | files changed | lines added | lines removed | |---|---|---|---| | Tanka | 3 | 29 | 0 | | cdk8s | 1 | 16 | 5 | | Kapitan | 3 | 38 | 2 | All three reuse their template, so api is configuration only. cdk8s is smallest partly through layout: its per-environment values are one object literal per line, the Jsonnet and YAML files use one line per field. Helm needed 25 lines here and Kustomize 149. A value derived from another All three compute web_url from the web Service port. In a copy I changed only that port to 8081 in each tool. check still passed, and all three rendered web_url: http://web.shop.svc:8081. In the same test Helm and Kustomize rendered the stale :80 without an error. For Tanka vs Helm and cdk8s vs Helm this is the main difference: a language lets one value reference another, separate Helm values files cannot. The mechanisms differ. Jsonnet binds $._config late, so an environment that overrides the web port also changes the api URL. In cdk8s it is an ordinary variable, so the dependency follows your code structure. Kapitan interpolates ${...} across the whole inventory, so any class or target can reference any value in another file. Typos and type safety I misspelled pdb_min_available as pdb_min_availabe in the prod web values of each tool: | | result | |---|---| | Tanka | renders, exit 0, web PodDisruptionBudget missing | | cdk8s | synth fails: error TS2561: ... 'pdbMinAvailabe' does not exist in type 'Sizing'. Did you mean to write 'pdbMinAvailable'? | | Kapitan | compiles, exit 0, web-scaling.yaml missing | cdk8s catches it because the example declares an interface for its values, and the generated imports type every Kubernetes field for the editor. Jsonnet has no types: reading a missing field fails, adding a misspelled one does not, unless you write asserts. jsonnet-language-server adds completion. Kapitan's inventory is YAML, and neither Kapitan nor the generator rejected the unknown key. An output diff in CI, like check, caught it in all three. Toolchain and speed Install is one run with cold caches and depends on the network. Node.js 26 and Python 3.13 were already present and are not counted. Render covers both environments, five warm runs. | | install | on disk | render | |---|---|---|---| | Tanka | 12.1 s for tk and jb, 3.4 s jb install | 29 MB + 7.3 MB | 0.10 to 0.11 s | | cdk8s | 4.2 s npm ci, 173 packages | 95 MB | 3.46 to 3.59 s | | Kapitan | 63.1 s uv run, 89 packages; 13.3 s generator verify and pull | 285 MB + 6.2 MB | 1.61 to 1.67 s | Tanka is two static Go binaries and by far the fastest. Much of the cdk8s time is ts-node type-checking the 41214-line import file: transpile-only, the same synth took 2.27 to 2.36 s. Kapitan reports 0.68 s for the compile, the rest is uv run and Python startup. Its install is large because the base package depends on cloud SDKs for its secret backends: google-api-python-client alone is 112 of the 267 MB installed. Reuse and packages Tanka libraries are Git repositories that jsonnet-bundler pulls and locks by commit, with no central registry. jsonnet-bundler had no commits in the last 12 months and its latest release, v0.6.0, dates from August 2024. cdk8s uses npm or the package manager of its other languages, and Construct Hub indexes published constructs. The kadet-hub catalog lists 36 Kapitan generators, each an OCI artifact with a build attestation. The registry is young: created on 2026-10-08, one human committer so far. All three consume Helm charts: Tanka through tanka-util, cdk8s through cdk8s import helm: and its Helm construct, Kapitan through its helm input type. Kapitan and Jsonnet Kapitan has a jsonnet input type, so the choice is less exclusive than it looks. I pointed a Kapitan target at the unchanged Tanka library and the vendored k8s-libsonnet, with an 11-line main.jsonnet that reads the prod values from the inventory. After the same normalization the output matched tk show environments/prod, all 8 objects. A team with a Jsonnet codebase can keep it and add inventory, refs and non-Kubernetes outputs, giving up tk diff and tk apply against a cluster. Output beyond Kubernetes Tanka and cdk8s render Kubernetes objects. Plain Jsonnet can emit Terraform's JSON syntax, outside Tanka. CDKTF, the Terraform counterpart of cdk8s, was archived on 10 December 2025 with no further updates. Kapitan compiles any output from one inventory, and kadet-hub lists Terraform generators for AWS, AzureRM, Google Cloud and Open Telekom Cloud, all at 0.1.x. GitOps Argo CD renders Helm, Kustomize and plain Jsonnet directories natively, Flux renders Helm and Kustomize. None of the three tools is native to either. A Tanka environment does not fit Argo CD's Jsonnet support: tk eval returns nested objects (apps.web.deployment) that Tanka flattens, and the namespace comes from spec.json. The usual routes are an Argo CD config management plugin that runs the tool in the repo server, or rendering in CI and committing the YAML or pushing it with flux push artifact. Tanka has tk export for that, and Kapitan always writes files. cdk8s has one more option: cdk8s synth --format helm writes a chart with Chart.yaml and templates, which both Argo CD and Flux deploy natively. Project activity Read-only GitHub API on 2026-10-17, window from 2025-10-17. Human commits exclude bot and automation accounts. | repository | stars | commits | human commits (authors) | releases | latest | |---|---|---|---|---|---| | grafana/tanka | 2690 | 500 | 41 (14) | 17 | v0.39.4, 2026-10-08 | | cdk8s-team/cdk8s | 4859 | 499 | 7 (4) | 0 | umbrella repo | | cdk8s-team/cdk8s-core | 87 | 578 | 0 | 90 | v2.70.111, 2026-10-12 | | cdk8s-team/cdk8s-cli | 41 | 479 | 8 (4) | 125 | v2.207.67, 2026-10-13 | | cdk8s-team/cdk8s-plus | 163 | 635 | 8 (6) | 330 | cdk8s-plus-37 v2.0.0, 2026-10-16 | | kapicorp/kapitan | 1931 | 135 | 93 (11) | 6 | v0.36.3, 2026-07-03 | | kadet-hub/registry | 1 | 716 | 711 (1) | 5 | index-47, 2026-10-16 | 441 of Tanka's 500 commits are Renovate updates. All 578 cdk8s-core commits were automated dependency upgrades; its frequent releases carry those. Kapitan releases less often and has the most human commits of the three tools. Running it You need tk and jb, Node.js with npm, and uv, plus oras and a logged-in gh for the first generator fetch: $ git clone https://github.com/kadet-hub/examples $ cd examples/config-tools $ ./check The first run installs dependencies from the lock files and verifies the generator's attestation before pulling it by digest; the companion post covers what that fetch does not check. Originally published at kadet-hub.org.

Dockerのビルドを遅くしたコミットをGrafana Tempoのトレースで見つける ! 2026年10月20日に確認した内容です。動作確認には次のバージョンを使いました。 Docker Engine 29.0.0、buildx v0.29.1（Ubuntu 26.04.1） BuildKit v0.33.1（builderのイメージをmoby/buildkit:v0.33.1に固定） grafana/otel-lgtm 0.37.0（Grafana 13.2.3、Tempo 3.1.0を含む） mcp-grafana 2.0.2 Claude Code 2.1.303 gcx 0.3.0 記事中のコマンドは、手元で使ったポート番号をotel-lgtmのデフォルトに戻して書いています。 はじめに こんにちは、Grafana Labsでデベロッパーアドボケイトをしているものです。 2026年11月3日にシンガポールで開かれるGrafana & Friends Singapore x Dockerで話すことになりました。Dockerとの共催というせっかくの機会なので、題材には以前から話そうと思っていたDockerのビルドのトレースを選んでいます。Docker自身もビルドのトレースを画面に出せますが、Grafana Tempoに送って溜めておくと、ビルドをまたいだ調査が可能です。この記事は、その手順と、溜めたトレースをAIに調べさせた結果のまとめです。 TL;DR Dockerのビルドを実行するBuildKitは、ビルドごとにOpenTelemetryのトレースを出しています。そのトレースはdocker buildx history traceでJaeger UIに開けるので、ビルドを1回調べるならその画面で十分です。ただし、BuildKitが残す履歴はデフォルトで48時間か50回分までです。同じトレースをGrafana Tempoに送ってコミットのIDを付けておけば、どのコミットからビルドが遅くなったのかをグラフで特定できます。コーディングエージェントやGrafana Assistantにトレースを読ませれば、遅くしたコミットと手順を見つけるところまで任せられます。 BuildKitが出すトレース docker buildの実体は、buildxというCLIのプラグインと、ビルドを実行するデーモンのBuildKitです。BuildKitはOpenTelemetryでトレースを出す機能を持っていて、Dockerfileの手順ひとつひとつがスパンになります。計装を足す必要はありません。 Dockerの公式ドキュメントが示しているのは、環境変数JAEGER_TRACEでJaegerへ送る方法だけです。一方、BuildKitのREADMEには、OTLPで送る方法が載っています。環境変数OTEL_TRACES_EXPORTER=otlpとOTEL_EXPORTER_OTLP_TRACES_ENDPOINTを使う方法で、後でTempoへ送るときにもこちらを使います。 トレースをどこにも送っていなくても、BuildKitはビルドの履歴にトレースを残しています。docker buildx history traceは、その履歴からトレースを取り出し、一時的に立ち上げたJaeger UIで開くコマンドです1。 docker buildx history trace --builder gf-demo --addr 127.0.0.1:16686 docker buildx history traceで開いたJaeger UIの画面。上部はトレースの概要で、下はDockerfileの手順のスパンのあたりを切り出した この画面では、全体の20.27秒のうち19.24秒を[build 4/4] RUN go mod download && go buildの手順が占めていることがわかります。ビルドが遅いと感じたときに、どの手順を疑えばよいかは、この画面だけで判断できます。 Tempoへ送信 同じトレースをGrafana Tempoへ送るには、buildxのbuilderを作るときに環境変数を2つ渡すだけです。送り先にはgrafana/otel-lgtmを使いました。GrafanaとTempoに加えてPrometheusとOpenTelemetry Collectorも1つにまとめたコンテナで、そのOTLPのポートへ送ります。 docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 grafana/otel-lgtm:0.37.0 docker buildx create --name gf-demo --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt network=host \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=127.0.0.1 \ --bootstrap docker buildx build --builder gf-demo --load . docker-containerドライバーのbuilderでは、BuildKitはコンテナの中で動きます。network=hostを付けたのは、そのBuildKitからホストのループバックアドレスに公開したOTLPのポートへ届かせるためです2。 トレースには、buildx（サービス名buildx）とBuildKit（サービス名buildkitd）のスパンが1本につながって入ります。ルートのスパンはbuild .です3。クライアント側のbuildxには環境変数を渡していませんが、buildxのスパンもTempoに届きます4。 同じ種類のビルドのトレースをGrafanaのExploreで開いた画面。上部はトレースの概要で、下はDockerfileの手順のスパンのあたりを切り出した ビルドを遅くしてみる ここまでなら、Dockerの画面で十分です。Tempoへ送る価値が出てくるのは、ビルドが1回ではなく、CI/CDのように継続的に多数行われるときです。 BuildKitの設定では、ビルドの履歴はデフォルトでmaxAge = 172800（48時間）、maxEntries = 50です。docker buildx history traceで開けるのは、この履歴に残っているビルドだけで、--compareで2回分を並べて比べる場合も同じ制約がかかります。 使い捨てのCIのrunnerでは、ジョブが終わればbuilderごと履歴も消えます。DockerはGitHub Actions向けに、ビルドの記録をZIPのアーティファクトとして残す機能を持っています。ただ、それをDocker Desktopに取り込んで見るのは、CIの実行1回分ずつです。「どのコミットからビルドが遅くなったのか」を答えるには、何十回分ものビルドを取得して比較しなければなりません。 そこで、コミットを重ねながら何度もビルドし、全部のトレースをTempoに溜めました。題材は、Prometheusのクライアントライブラリに依存する小さなGoのHTTPサーバーです。はじめは、依存のダウンロードとビルドのキャッシュをキャッシュマウントで再利用するDockerfileでビルドしました。 FROM golang:1.26-alpine AS build WORKDIR /src COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod \ go mod download COPY . . RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build -o /out/hello . FROM gcr.io/distroless/static-debian13 COPY --from=build /out/hello /hello ENTRYPOINT ["/hello"] 途中で、Dockerfileを「整理した」コミットを入れて、キャッシュマウントをわざと外しました。あわせてCOPY . .も依存のダウンロードより前に移した状態です。この順序だと、ソースを1行変えるたびに依存のダウンロードと全パッケージのコンパイルがやり直しになります。buildのステージは以下のように変わっています。 FROM golang:1.26-alpine AS build WORKDIR /src COPY . . RUN go mod download && go build -o /out/hello . 前後には、Dockerfileにラベルを足すだけのコミットを1つずつ混ぜました。git logにはDockerfileを変えたコミットが3つ並ぶので、どれでビルドが遅くなったのかはトレースを見ないとわかりません。 各ビルドには、そのときのコミットのIDを付けました。buildxはOTEL_RESOURCE_ATTRIBUTESを読むので、ここにリソース属性vcs.ref.head.revisionとして入れて渡せば済みます。ただし、この属性が付くのはbuildxのスパンだけで、buildkitdのスパンは対象外です。 OTEL_RESOURCE_ATTRIBUTES="vcs.ref.head.revision=$(git rev-parse --short HEAD)" \ docker buildx build --builder gf-demo . ここまでの各ビルドの所要時間を、TraceQLのメトリクスのクエリでグラフにしました。ルートのスパンだけを選んで、30秒ごとの最大値を取っています5。 { resource.service.name = "buildx" && span:parentID = "" } | max_over_time(duration) 各ビルドの所要時間をTraceQLのmax_over_timeでGrafanaに描いた画面。点の時刻は30秒の区間の終わりを表す トレースで見ると、キャッシュマウントを外す前のビルドは1.9〜2.5秒で終わり、外した後は20.3〜21.7秒に延びて戻りません6。跳ねた直後のトレースを開けば、vcs.ref.head.revisionからコミットがわかります。このデモでは「整理した」コミットでした。遅くなる前後のトレースを並べると、どの手順が延びたのかもわかります。 遅くなる前のビルドのトレースをGrafanaのExploreで開いた画面。依存のダウンロードはキャッシュから読まれ、go buildは約0.9秒で終わっている 遅くなる前は、依存のダウンロードの手順がload cache:で始まるスパンになり、キャッシュから読まれています。遅くなった後は、手順がRUN go mod download && go buildの1つにまとまり、その手順だけで19.36秒かかっています。 手順のスパン名には、RUNのコマンドがそのまま入ります。Dockerfileを変えるとスパン名も変わるので、変更の前後を1本の系列で比べるなら、ルートのスパンの所要時間を使ってください。 エージェントに修正させる Tempoに溜めたトレースは、人だけでなくプログラムからも問い合わせられます。mcp-grafanaは、GrafanaをMCPのサーバーとして公開するツールです。Tempo向けには次のようなツールを持っています。 search_tempo_traces：TraceQLでトレースを検索する query_tempo_metrics：TraceQLのメトリクスを計算する diff_tempo_traces：2本のトレースを比べる これをコーディングエージェントにつなぎ、ビルドが遅くなった原因を調べさせました。mcp-grafanaの起動方法とMCPのクライアントへの登録方法は、mcp-grafanaの公式ドキュメントを参照してください。書き込み系のツールは--disable-writeで外せるので、調査に使うだけなら外しておくと安心です。 今回はClaude Codeを使いましたが、MCPのサーバーを使えるほかのコーディングエージェント（Codexなど）でも同じようにできるでしょう。Gitの履歴の参照やdocker buildx buildの実行など、必要なツールは適宜許可してください。頼んだ内容は次のとおりです。 コンテナのビルドが最近遅くなりました。ビルドのトレースはTempo（Grafanaのデータソース UID "tempo"）に送っています。各ビルドにはサービス buildx の "build ." という名前のスパンがあり、そのスパンのリソース属性 vcs.ref.head.revision にビルドしたコミットが入っています。Tempoのトレースを使って、どのコミットでなぜビルドが遅くなったのかを調べ、このディレクトリのDockerfileを直してください。直したら docker buildx build --builder gf-demo --output type=cacheonly . で再ビルドし、新しいビルドが速くなったことをTempoで確かめてください。新しいトレースが検索できるようになるまで1〜2分かかることがあります。 エージェントは、まずGitの履歴を読み、search_tempo_tracesでコミットごとの所要時間を並べました。そこから導いた結論は、所要時間が約2秒から約21秒に跳ねたのは「整理した」コミットだというものです。遅くなる前と後のトレースをdiff_tempo_tracesで比べ、RUN go mod download && go buildの手順に約19.9秒かかるようになったことと、以前は依存のダウンロードがキャッシュから読まれていたことも確かめました。 修正では、go.modとgo.sumを先にコピーする順序とキャッシュマウントを元に戻し、後から足されたラベルは残しました。再ビルドのあとは新しいトレースが検索できるようになるのを待ち、所要時間が2.31秒に戻ったことをTempoで確かめています。全体で約3分かかり、そのうち約1分半は新しいトレースを待つ時間でした。 オブザーバビリティのSaaSに送る利点 ここまでは、手元のotel-lgtmにトレースを送ってきました。otel-lgtmは、イメージの説明にあるとおり開発、デモ、テストのための環境で、コンテナを消せばトレースも消えます。ビルドをまたいで問いを立てるには、トレースを長く残せて、CIのrunnerからも送れる場所が必要です。Grafana CloudのようなオブザーバビリティのSaaSに送ると、その場所を自分で運用せずに済みます。たとえばGrafana Cloudでも、BuildKitで保持する量よりも圧倒的に多くのトレースを保持できます7。 Grafana Cloudへ送るときも、builderに渡す環境変数を変えるだけです。Grafana CloudのOTLPのゲートウェイにはOTLP/HTTPで送るので、BuildKitのデフォルトのgRPCからhttp/protobufに切り替えます。エンドポイントと認証のヘッダーは、Grafana CloudのポータルのOpenTelemetryのタイルで生成した値をそのまま使いました。 docker buildx create --name gf-cloud --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces" \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}" \ --bootstrap トレース専用の変数（名前にTRACESが入るもの）だけを渡しているのは、BuildKitがOTEL_EXPORTER_OTLP_ENDPOINTを見るとメトリクスの送信も始めるためです8。このbuilderで手元と同じ20回のビルドを流すと、21本（動作確認の1回を含む）のトレースが全部Grafana CloudのTempoに届きました9。トークンはbuilderのコンテナの環境変数とbuildxの設定ファイルに残るので、使い終わったらdocker buildx rm gf-cloudでbuilderを消してください。 Grafana Assistantに調査させる SaaSの側には、溜めたトレースを調べる機能も最初から用意されています。その例が、Grafana Cloudに組み込まれたAIアシスタントのGrafana Assistantです。ドキュメントによると、AssistantはTempoを含むデータソースに問い合わせ、TraceQLのクエリを書いて実行できます。トレースのIDを渡せば1本のトレースをウォーターフォールで表示でき、実験的な機能として、あるトレースと比べる候補のトレースを探すこともできます。 前の節でエージェントに調べさせたときは、mcp-grafanaを起動してエージェントに登録する準備が必要でした。Assistantは最初からスタックのデータソースにつながっているので、この準備は要らないのが利点です。ビルドが遅くなったことに気づいた人が、Grafanaの画面からそのまま「ビルドの所要時間はいつから延びたか」と聞けば済みます。原因の調査まではAssistantに任せ、Dockerfileを直して再ビルドするところはリポジトリを編集できるコーディングエージェントに任せる、という分担です。 実際に、Grafana Cloudへ送った20回分のビルドについてAssistantに聞きました。AssistantはGrafanaの画面のほか、Grafanaのコマンドラインツールのgcxからも呼べるので、今回はgcxを使いました。伝えたのは、コーディングエージェントに渡したのと同じ前提だけです。 gcx assistant prompt 'コンテナのビルドが最近遅くなりました。ビルドのトレースはTempo（データソース grafanacloud-shophub-traces）に送っています。各ビルドにはサービス buildx の "build ." という名前のスパンがあり、そのスパンのリソース属性 vcs.ref.head.revision にビルドしたコミットが入っています。過去1時間のトレースから、ビルドの所要時間がいつから延びたのか、どのコミットからなのか、ビルドのどの手順が遅くなったのかを調べてください。' Assistantはデータソースへの問い合わせを17回行い、132秒で答えを返しました。答えの冒頭は次のとおりです。 結論: ビルド時間は、およそ 07:09 UTC に約2秒から約20秒へ延びました。最初に遅くなったコミットは 9eac8b9 です。遅くなった手順は Dockerfile の RUN go mod download && go build -o /out/hello . です。 9eac8b9は「整理した」コミットで、答えは正しいものでした。手順についても、速いビルドの6ステップと遅いビルドの4ステップを比べた結果、遅いほうではRUN go mod download && go buildの1つの手順が約19.4秒、全体の約95%を占めるという答えです。依存の取得のキャッシュが効かなくなったことを原因として推測しつつ、「このデータにはコミットの親子順がありません」と書き、時間の順で最初に遅くなったコミットだという限定も付けていました。動作確認のために送った30.2秒のビルドも、コミットの属性がないことに気づいて、ほかのビルドとは別に扱っていました。 Assistantは、自前で運用しているGrafanaからGrafana CloudのAssistantにつないで使うこともできます10。 おわりに Dockerは、ビルドのトレースを出す仕組みも、それを確認する画面も持っています。ビルドを1回調べるならDockerの画面で十分で、Tempoに送る意味が出てくるのはビルドをまたいだ問いを立てるときだと確認しました。 この記事では、長くなるので、ビルドのトレースをGrafanaに送り、溜めたトレースをAIに調べさせるところまでにしました。次は、これを踏まえてCIの設定も含めて書きたいと考えています。CIのジョブがTRACEPARENTを渡せば、ビルドのスパンをジョブのスパンの下に入れて、パイプライン全体の中でビルドを見られるはずです。 ビルドが遅くなったと感じたことがあるなら、まず手元のbuilderに環境変数を2つ足して、トレースを溜めるところから始めてみてください。コンテナビルドでもオブザーバビリティを高めていきましょう。 脚注 標準出力が端末でないときは、Jaeger UIを起動せずに、トレースをJaegerの形式のJSONとして標準出力に書き出しました。 ↩︎ 確認したのはLinuxのDocker Engineだけです。Docker Desktopではnetwork=hostの意味が変わるので、この設定のままでは届かない可能性があります。 ↩︎ buildxは、環境変数TRACEPARENTがあると、それを親のトレースコンテキストとして扱います。この変数が設定されたシェルからビルドすると、ビルドがその親のトレースに入り、build .がルートになりません。ビルドを1回ずつ別のトレースにしたいなら、TRACEPARENTを外してから実行してください。buildx v0.29.1は、cmd/buildx/tracing.goでBuildKit v0.25.0のutil/tracing/envを読み込んでいます。このパッケージがTRACEPARENTとTRACESTATEを読みます。buildxのもっと新しいバージョンでは、同じ役割のパッケージがutil/tracing/childprocessに変わっています。 ↩︎ buildx v0.29.1は、BuildKitに接続するときにclient.WithTracerDelegateを指定しています（driver/manager.go）。buildxは自分のスパンをBuildKitに渡し、BuildKitがまとめて送ります。BuildKitのREADMEにも、buildctlはスパンをbuildkitdに転送するので、exporterはデーモン側だけに設定すればよいとあります。また、docker-containerドライバーではbuildctl dial-stdioというトレースがビルドとは別に毎回届くので、検索するときはresource.service.name = "buildx"で絞ると見やすくなります。 ↩︎ quantile_over_time(duration, .5)で中央値を取ると、約20秒のビルドが、区間を60秒にすると24.3秒、30秒にすると34.4秒になります。quantile_over_timeは所要時間を2のべき乗の幅のバケットに入れて近似するので、値がバケットの境目に寄ります。max_over_timeは、区間に入ったトレースの所要時間の最大値をそのまま返します。 ↩︎ グラフの範囲から外した最初のビルドは、キャッシュが空だったので38.1秒かかりました。 ↩︎ Grafana Cloudの料金のページによると、Tracesの保持期間はFreeプランで14日、Proプランで30日です。BuildKitの履歴の48時間か50回分より長く、手元の開発機のビルドとCIのビルドを同じTempoに入れて、同じクエリで比べられます。 ↩︎ BuildKit v0.33.1のutil/tracing/detect/otlp.goは、OTLPのクライアントをオプションなしで作っています。OpenTelemetryのGoのexporterは、送り先、プロトコル、ヘッダーを環境変数から読みます。 ↩︎ Grafana CloudのTempoでビルドのトレースを検索するときは、name = "build ."で絞ると確実に見つかります。 ↩︎ Grafana Assistantの紹介のページによると、自前で運用しているGrafanaから使う場合も、チャット、ダッシュボードの作成、クエリ、ナビゲーションは使えます。ただし、複数のデータソースを調べてレポートにまとめるInvestigationsと、Grafana CloudのMCPへの接続は、Grafana Cloudに限られた機能です。 ↩︎

Your CI runs other people's code. Is anyone still maintaining it? --- title: "Your CI runs other people's code. Is anyone still maintaining it?" published: true description: "Workflow scanners check how you use GitHub Actions. Few people check whether each Action is still maintained. What we found in Grafana's workflows — and how they fixed it." tags: security, opensource, github, devops cover_image: --- Every third-party uses: line in a GitHub Actions workflow runs someone else's code in your job. Depending on the workflow, that code can reach your GITHUB_TOKEN, the secrets passed to that step, and your checked-out source. In March 2025, the widely used tj-actions/changed-files Action was compromised: its tags were pointed at a malicious commit that printed CI secrets into workflow logs (CISA alert, CVE-2025-30066). That project was actively maintained — a maintenance check would not have prevented it — and a fixed release followed. That last part is the point of this post. When something goes wrong in an Action that is still maintained, a fix comes out and you upgrade. When the Action is no longer maintained, there is no next release. The question workflow scanners don't ask There are good tools for securing your workflows: | Tool | What it checks | |---|---| | OpenSSF Scorecard | Your repo's practices: dangerous workflow patterns, token permissions, pinned dependencies | | zizmor | Your workflow files: injection risks, pull_request_target misuse, and more | | actionlint | Your workflow syntax and expressions | All three look at your workflow. Scorecard does have a Maintained check, and it gives an archived repository the lowest score — but only for the repository you run it on. Few teams run Scorecard against each of the 30-odd Actions their workflows reference. Pinning Actions by commit SHA is good practice. It stops a tag from being silently repointed. But it doesn't tell you whether anyone will ship a fix for the code you pinned. !Three layers of code run in your job: your workflow file, the Actions its uses: lines reference, and Actions called inside composite Actions. Scorecard, zizmor and actionlint read the first layer. uzomuzo with --include-actions checks the repository behind each referenced Action, such as the archived tibdex/github-app-token in Grafana's release-build.yml; --show-transitive adds the third layer. You can fix the second layer by changing a uses: line; the third needs the parent Action's maintainer. uzomuzo does that inventory for you: it reads your workflows, lists every Action they reference, and reports the maintenance status of each. It complements the tools above — it doesn't replace them. What we found in Grafana (April 2026) Grafana is one of the most popular open-source projects on GitHub. In April 2026 we scanned the Actions referenced in its workflows. One of them was archived: tibdex/github-app-token. - The repository was archived in July 2025 (its last push is July 7, 2025). Its README says the Action is deprecated and points to GitHub's official actions/create-github-app-token — the maintainer did exactly the right thing. - Grafana used it in 14 places across 13 workflow files, including release-build.yml and release-npm.yml. - It declares runs.using: node20. GitHub removed Node 20 from Actions runners on September 23, 2026, and an archived Action can't be updated to Node 24. Anyone still using it after that date has a broken workflow. An Action runs in your job, with access to its workspace, not in an isolated sandbox. Depending on the workflow, it can reach GITHUB_TOKEN (with whatever permissions the job grants), the secrets passed to it, and the checked-out source. Release jobs often hold sensitive credentials. That doesn't mean anything was exposed here — it means an archived Action in a release job deserves a look. What happened next We filed an issue on April 5, 2026. Grafana had already begun moving to actions/create-github-app-token in one workflow in October 2025. A community member opened a PR for the remaining files three days after our issue. A maintainer closed it, explaining that the team had a new recommended approach using their own internal workflow. We don't know whether our issue influenced their timing. On September 15, 2026, the issue was closed as completed: "This has been done, we no longer use that action :)" — a week before Node 20 was removed. When we scanned Grafana's workflows again on October 26, 2026, tibdex/github-app-token no longer appeared, and none of the 32 Actions was archived or got a replace verdict. Reporting doesn't always lead to a fix. Of the four upstream reports we filed in April, two have been resolved and two have had no response yet. Whether and how to fix something is the maintainers' call. What a scanner can do is make sure the question gets asked. Scan your own Actions Install go install github.com/future-architect/uzomuzo-oss/cmd/uzomuzo@latest Or download a binary from GitHub Releases. You need v0.8.0 or later for --fail-on archived and the · archived marker. The examples below assume GITHUB_TOKEN is set: scanning a repository's Actions reads workflow files through the GitHub API, and the token also gives uzomuzo the repository signals (archived, commit history) it needs. Check a repository export GITHUB_TOKEN=... # a read-only token is enough # Actions referenced directly in your workflows uzomuzo scan https://github.com/your-org/your-repo --include-actions # Also include Actions used inside composite Actions uzomuzo scan https://github.com/your-org/your-repo --include-actions --show-transitive A composite Action can itself uses: other Actions. Those run in your job too. The difference is who can fix them: you can swap a direct Action today, but a transitive one needs the parent Action's maintainer to act. You can also scan a single Action or workflow file: uzomuzo scan https://github.com/tibdex/github-app-token uzomuzo scan --file .github/workflows/ci.yml What an archived Action looks like $ uzomuzo scan https://github.com/tibdex/github-app-token https://github.com/actions/github-script STATUS PURL LIFECYCLE BUILD 🔴 replace https://github.com/tibdex/github-app-token Stalled · archived — ⚠️ caution https://github.com/actions/github-script Stalled — Both Actions are Stalled: neither has had much recent activity. Only one is archived, and that's the one marked · archived and given a replace verdict. The lifecycle label stays Stalled rather than EOL-Confirmed on purpose. uzomuzo uses EOL-Confirmed only for an explicit end-of-life signal, such as an npm deprecation or a Packagist "abandoned" flag. Archiving alone isn't one: some archived repositories are packages that moved to a monorepo and keep shipping releases from there. So the archive is shown as a separate fact next to the label instead. (tibdex's README does say "This action is deprecated", but uzomuzo deliberately doesn't treat README prose as a signal — free text is easy to misread.) If you read the earlier post in this series: its table showed archived repositories as EOL-Confirmed. Current versions show them as above. Gate your CI uzomuzo scan https://github.com/your-org/your-repo --include-actions --fail-on archived --fail-on archived fails the build when any referenced Action's repository is archived — and, like every --fail-on value, when a version is known to be malicious. It doesn't fire on Actions that are merely quiet. That matters: --fail-on stalled would also catch archived Actions, but in our October scan of Grafana it would have failed on four Actions that are simply less active, including actions/github-script. --fail-on eol-confirmed alone wouldn't fail on an archived Action at all. Start with a report-only run, look at what comes back, then turn the gate on. Run it on a schedule The uzomuzo repository has a monthly workflow that generates an SBOM with aquasecurity/trivy-action, runs uzomuzo scan --sbom, and opens an issue with the results — SCA and lifecycle checks side by side. That workflow covers package dependencies, not Actions. To cover Actions as well, add a step that runs uzomuzo scan https://github.com/$GITHUB_REPOSITORY --include-actions --fail-on archived with GITHUB_TOKEN set. Two layers, two questions Scorecard, zizmor, and actionlint answer: "Is my workflow configured safely?" uzomuzo answers: "Is the code my workflow runs still maintained?" A safely configured workflow can still depend on an Action that will never get another release. That isn't a known vulnerability. It's something to review before it becomes a broken build — or worse. --- Earlier post: "Your dependencies are 48% unmaintained — and SCA tools can't see it" Source and docs: future-architect/uzomuzo-oss (Apache 2.0) Kota Kanbe — Creator of vuls, an open-source vulnerability scanner.