signoz

SigNoz: The Open-Source Observability Platform That Actually Replaces Datadog SigNoz: The Open-Source Observability Platform That Actually Replaces Datadog You've probably heard "SigNoz is the open-source Datadog alternative" in every other dev post. But most explanations stop at "it uses OpenTelemetry and ClickHouse" -- as if that explains why it works. It doesn't. The real difference isn't the tech stack. It's how SigNoz solves the fundamental problem every observability platform faces: how to store and query logs, metrics, and traces together without melting your infrastructure or your budget. The Problem Everyone Lives With If you've used Datadog, New Relic, or any major SaaS observability platform, you know the drill: - Logs go to one system (Elasticsearch, Loki, CloudWatch) - Metrics go to another (Prometheus, Datadog metrics, InfluxDB) - Traces go to a third (Jaeger, Zipkin, Tempo) - Correlation happens in your head -- you switch tabs, copy trace IDs, pray the timestamps align This isn't an accident. It's the natural consequence of how these systems evolved. Logs need full-text search. Metrics need time-series aggregation. Traces need span trees and latency percentiles. Three different query patterns. Three different storage engines. Datadog solves this by charging you for three different products and gluing them together in the UI. You pay per host, per custom metric, per log ingested, per trace retained. The bill grows faster than your traffic. SigNoz takes a different bet: one columnar database for everything. Why ClickHouse Changes the Game ClickHouse isn't just "fast." It's a column-oriented OLAP database built for analytical queries over massive datasets. Here's why that matters for observability: One Query Language for All Signals In SigNoz, a trace, a log line, and a metric live in the same database. You can write a single SQL query that joins spans with logs and aggregates metrics: -- Find all errors in the payment service, show related logs, and calculate error rate SELECT toDateTime(ts_bucket_start) as time, countIf(has_error = 1) as error_count, count() as total_spans, error_count / total_spans as error_rate FROM signoz_index_v3 WHERE service_name = 'payment-service' AND ts_bucket_start >= now() - INTERVAL 1 HOUR GROUP BY time ORDER BY time DESC Try doing that across Elasticsearch + Prometheus + Jaeger. You can't. You'd need application-level correlation, which means more code, more latency, more things to break. High-Cardinality Without the Tax This is the killer feature nobody talks about. In Prometheus/Datadog, every unique label combination creates a new time series. Add request_id, user_id, session_id to your metrics and you've created a cardinality explosion. Your bill explodes. Your query latency explodes. You're told "don't do that." In ClickHouse, high-cardinality attributes live in typed Map columns. They're compressed with ZSTD. They don't create new time series. You can filter by user_id across billions of spans. It just works. SigNoz takes this further: hot attribute keys get promoted to materialized columns automatically. The system adapts to your query patterns. Compression That Actually Matters ClickHouse uses per-column codecs tuned for observability data: | Data Type | Codec | Why It Works | |-----------|-------|--------------| | Timestamps | DoubleDelta | Monotonic increasing = tiny deltas | | Strings (labels, trace IDs) | ZSTD | Dictionary + entropy coding | | Metric values (latencies, counts) | Gorilla | XOR-based floating-point compression | | Integers (durations, counts) | T64 | Variable-byte encoding | Typical observability datasets compress 10-20x. That's not marketing -- it's columnar storage doing what it was designed for. The Architecture: Three Planes, Not Five Microservices SigNoz used to ship five separate containers: query-service, frontend, alertmanager, otel-collector, ClickHouse. That was v0.75 and earlier. Since v0.76 (March 2025), it's three components: +---------------------------------------------------------------+ | SigNoz Binary (single Go binary) | | +----------+ +----------+ +----------+ +------------------+ | | | React | | API | | Ruler | | Alertmanager | | | | UI | | Server | | (alerts)| | (notifications) | | | +----------+ +----------+ +----------+ +------------------+ | | | | | +-----+-----+ | | | OpAMP | <-- Manages collector config | | | Server | | | +-----------+ | +---------------------------------------------------------------+ | v +---------------------------------------------------------------+ | SigNoz OTel Collector (ingestion plane) | | OTLP gRPC/HTTP * Jaeger * Zipkin * Prometheus * Kafka | | Custom processors: span metrics, span mapper, LLM pricing | +---------------------------------------------------------------+ | v +---------------------------------------------------------------+ | ClickHouse (+ Keeper/ZooKeeper) | | signoz_traces * signoz_logs * signoz_metrics * signoz_meter | | Resource fingerprint tables for fast filtering | +---------------------------------------------------------------+ The Ingestion Plane: SigNoz OTel Collector This isn't a vanilla OpenTelemetry Collector. It's a curated distribution with SigNoz-specific components: Receivers: OTLP (gRPC 4317, HTTP 4318), Jaeger, Zipkin, Prometheus, Kafka, OpenCensus, HTTP logs (Heroku, JSON) Processors that matter: - signozspanmetrics/delta -- computes RED metrics (Rate, Errors, Duration) from spans automatically. No separate metrics pipeline needed. - signozspanmapper -- normalizes framework-specific attributes (LangChain, LlamaIndex, etc.) into standard gen_ai.* semantic conventions - signozllmpricing -- multiplies token counts by per-model prices, writes cost attributes per span - batch -- 50k spans / 5 seconds (ClickHouse best practice: large, infrequent inserts) Exporters: ClickHouse (traces, logs, metrics, meter), metadata exporter (drives query builder autocomplete) Connector: signozmeter -- counts ingested volume per service/environment/host, powers usage dashboards Extension: OpAMP health check + managed config. The collector connects to the OpAMP server inside the SigNoz binary, receives its config remotely, and applies it without redeployment. Log pipelines, span mapper rules, LLM pricing -- all editable in the UI. The Storage Plane: ClickHouse Schema Design This is where most observability platforms struggle. SigNoz's schema is worth understanding because it explains the query performance. Traces (signoz_index_v3) and Logs (logs_v2) share the same layout: Main table: Resource table (companion): +-----------------+ +---------------------------+ | ts_bucket_start | | resource_fingerprint (PK) | | resource_fp ----+-->| labels (string map) | | trace_id | | seen_at_ts_bucket_start | | span_id | +---------------------------+ | duration_nano | | has_error | | ...attributes...| +-----------------+ Resource attributes (service.name, k8s.namespace.name, deployment.environment) are hashed into a resource_fingerprint. The fingerprint is stored once per time bucket in the companion table. The main table only carries the fingerprint + timestamp bucket. Query execution: 1. Filter resource table by attributes (e.g., service.name = 'payment-service') 2. Get matching fingerprints 3. Scan main table granules: resource_fingerprint GLOBAL IN (...) + ts_bucket_start range This is why SigNoz docs tell you to always filter on resource attributes first. Without a resource filter, you scan all granules. With it, you scan only the relevant ones. Metrics (signoz_metrics_v2) use a different layout optimized for time-series: metric name + labels as columns, value + timestamp. PromQL and ClickHouse SQL both work. TTL strategy: Daily partitions, ttl_only_drop_parts = 1. Retention drops whole parts (15 days for logs/traces, 30 days for metrics) -- no row-level deletes, no merge overhead. What You Actually Get: Feature Walkthrough APM That Works Out of the Box Instrument your app with OpenTelemetry (auto-instrumentation exists for Java, Python, Node, Go, .NET, PHP, Ruby). Send to SigNoz OTel Collector. You get: - Service map with latency/error/throughput per service - Top endpoints with p50/p90/p99 latency, error rate, Apdex - Database calls, external calls, message queue spans -- all correlated - No dashboard building required. The APM views are generated from span data. Logs That Correlate With Traces Click a trace span -> see all logs with the same trace_id. Click a log -> jump to its trace. This works because both share the resource fingerprint + time bucket layout. No separate correlation engine needed. Metrics From Spans (No Separate Pipeline) The signozspanmetrics processor computes delta-temporality histograms and call counts per service/operation every 60 seconds. Dimensions: service.namespace, deployment.environment, service.version. Your APM dashboards are powered by span-derived metrics -- you don't need a separate Prometheus scrape setup for RED metrics. LLM/AI Observability Built In Since v0.143, the collector includes: - signozspanmapper -- normalizes input.value, gen_ai.prompt -> standard gen_ai.input.messages / gen_ai.output.messages - signozllmpricing -- reads gen_ai.usage.*_tokens, multiplies by model prices (pushed via OpAMP), writes signoz.gen_ai.usage.*.cost You get token usage, latency, cost per request, model breakdown -- all in the same traces view. Alerting That Doesn't Require a Separate Stack The ruler (inside the SigNoz binary) evaluates alert rules on a schedule. Queries run against ClickHouse. Alertmanager handles deduplication, grouping, notification routing (Slack, PagerDuty, OpsGenie, webhook, email). Threshold, anomaly, and Apdex-based alerts all supported. Dashboards: Query Builder, PromQL, or Raw ClickHouse SQL Three query modes: 1. Query Builder -- visual, no-code, generates ClickHouse SQL 2. PromQL -- for Prometheus migrants, runs via ClickHouse's PromQL-compatible engine 3. ClickHouse SQL -- full power for complex joins, cross-signal correlation Dashboard panels can mix all three. Variables, templating, drill-downs -- standard fare. Deployment: From Laptop to Production Local (Docker Compose) git clone https://github.com/SigNoz/signoz.git cd signoz/deploy/docker/clickhouse-setup ./install.sh # UI at http://localhost:8080 Bare Metal (systemd via Foundry) # Foundry = declarative CLI for self-hosted SigNoz # One config provisions infra, installs backend, sets up OTel collection foundry apply -f config.yaml Supports Docker, Swarm, systemd, Kubernetes, ECS, bare metal. Single binary + ClickHouse + Keeper. Kubernetes (Helm) helm repo add signoz https://charts.signoz.io helm install signoz signoz/signoz -n platform Production: replicated ClickHouse (3+ shards), 3-node Keeper ensemble, PostgreSQL metastore, multiple SigNoz replicas behind ingress. SigNoz Cloud Fully managed. 30-day free trial, usage-based pricing starting at $49/month. SOC 2 Type II, HIPAA, RBAC, SSO, BYOC option. The Trade-offs (Be Honest) SigNoz isn't magic. Here's what you're signing up for: | Advantage | Cost | |-----------|------| | No vendor lock-in -- OpenTelemetry native, your instrumentation is portable | You operate ClickHouse -- need fast local disks (NVMe), CPU for merges, Keeper/ZooKeeper quorum | | Predictable pricing -- no per-host, per-custom-metric, per-user-seat fees | Schema migrations -- upgrades require running the migrator job, occasional breaking changes | | Single database for all signals -- cross-signal correlation in SQL | Operational complexity -- more moving parts than a SaaS dashboard | | High-cardinality friendly -- filter by user_id, request_id freely | Metrics cardinality still costs -- unbounded labels on metrics create time series | | Self-hostable on bare metal -- no Docker/K8s required | Foundry learning curve -- new tool, replaces raw Compose files since v0.130 | The AGPL collector matters: The OTel Collector distribution is AGPL-3.0. If you modify and redistribute it, you must open-source your changes. Running it unmodified? No issue. Building a managed service on top? Talk to a lawyer. When SigNoz Makes Sense - You're already on OpenTelemetry or willing to adopt it (vendor-free instrumentation) - You want correlated logs/metrics/traces without stitching three SaaS products - High-cardinality attributes are part of your debugging workflow - You have infrastructure capacity (or budget for managed Cloud) to run ClickHouse - You prefer SQL/PromQL over proprietary query languages When to Stick With Datadog/New Relic - You only need infrastructure monitoring (host maps, cloud integration depth) - You don't want to operate a database -- SaaS wins on operational simplicity - You need Log Patterns / anomaly detection on logs -- SigNoz doesn't have this yet (roadmap) - Your team is small and observability isn't a core competency Quick Start: Instrument a Go Service in 5 Minutes // main.go package main import ( "context" "net/http" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/sdk/resource" "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" ) func initTracer() (*trace.TracerProvider, error) { exporter, err := otlptracehttp.New(context.Background(), otlptracehttp.WithEndpoint("localhost:4318"), otlptracehttp.WithInsecure(), ) if err != nil { return nil, err } tp := trace.NewTracerProvider( trace.WithBatcher(exporter), trace.WithResource(resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceName("go-demo"), semconv.DeploymentEnvironment("development"), )), ) otel.SetTracerProvider(tp) return tp, nil } func main() { tp, _ := initTracer() defer tp.Shutdown(context.Background()) http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) { ctx, span := otel.Tracer("go-demo").Start(r.Context(), "hello-handler") defer span.End() // Simulate work span.AddEvent("processing") w.Write([]byte("Hello, SigNoz!")) }) http.ListenAndServe(":8080", nil) } Run SigNoz locally (./install.sh in the docker compose dir), run this service, hit /hello a few times. Open http://localhost:8080 -> Traces -> go-demo. You'll see the span, latency, and if you add logs, they'll correlate automatically. The Bigger Picture SigNoz represents a shift in how observability is built: open standards + columnar storage + unified query layer instead of proprietary agents + fragmented storage + per-signal pricing. The industry is moving toward OpenTelemetry anyway. Datadog, New Relic, Grafana, Splunk -- all support OTLP ingestion now. But they still charge you for the storage and query layer they built around it. SigNoz says: the storage and query layer should be open too. --- Repository: SigNoz/signoz (32k+ stars) Docs: signoz.io Slack: signoz.io Try Cloud: signoz.io (30-day free trial) Tags: opensource, observability, opentelemetry, clickhouse, datadog-alternative, selfhosted

VictoriaLogs 与 Loki、Elasticsearch、OpenSearch、SigNoz、ClickStack 的日志写入性能对比 > This blog post is also available in English: [VictoriaLogs vs Loki, Elasticsearch, OpenSearch, SigNoz and ClickStack on structured log ingestion]( victoriametrics.com) 在 VictoriaMetrics 内部，我们会持续地对自己的产品进行性能测试。这能帮助我们保持竞争力，让每一个版本都变得更好。随着越来越多的团队开始使用 VictoriaLogs ，也有越来越多的人问我们：它适合哪些场景？切换过来能获得什么？因此，我们把基准测试作为一个[公开 GitHub 仓库]( VictoriaMetrics/logs-benchmark)发布出来，任何人都可以运行它来复现我们的结果，我们也会继续以开源的方式维护它。 这不是一篇劝你在迁移到 VictoriaLogs 的广告。相反，如果你在采集结构化日志（无论来自 Kubernetes 还是其他地方），本文应该能帮你选择合适的工具，并了解适合你的权衡取舍。 对不同系统进行比较的基准测试总是充满争议，因为每个系统擅长的场景各不相同。本文只测试一个场景：写入结构化日志，即每一行都带有命名字段的日志，Kubernetes 上的许多服务都是这样输出日志的。每个系统都运行在一台相同规格的机器上。查询性能、集群模式和对象存储会在后续文章中介绍。 仅凭写入速度无法决定该选哪个系统，所以本文也会深入它们的内部实现： - 为什么 Loki 写入速度最快，但一个简单的 Dashboard 查询却要花上几分钟？ - 为什么 SigNoz 和 ClickStack 落后于 VictoriaLogs ，尽管三者都以列式方式存储日志？ - 为什么 Elasticsearch 和 OpenSearch 需要那么多磁盘空间？ 这些问题的答案能帮助你规划硬件和成本，避免在日志量增长时遇到意外。 测试与环境 我们测试了截至 2026 年 10 月 6 个系统的最新版本，配置尽量接近默认值： - VictoriaLogs [v1.53.0]( github.com/VictoriaMetrics/VictoriaLogs/releases/…) - Loki [3.7.8]( github.com/grafana/loki/releases/tag/v3.7.8) - Elasticsearch [9.5.5]( github.com/elastic/elasticsearch/releases/tag/v9.…) - OpenSearch [3.9.0]( github.com/opensearch-project/OpenSearch/releases…) - SigNoz [v0.145.0]( github.com/SigNoz/signoz/releases/tag/v0.145.0) - ClickStack [2.39.1]( github.com/hyperdxio/hyperdx/releases/tag/%40hype…) 我们调整了少量配置，例如 Loki 的写入限制，以便每个系统都能在单节点上达到其设计上限。VictoriaLogs 不需要任何调整，因为它的设计目标就是开箱即用地提供最佳性能（[文档]( docs.victoriametrics.com)）。[仓库]( github.com/VictoriaMetrics/logs-benchmark/tree/ma…)中说明了每一项改动。 为了让测试保持简单，每个系统都以单节点模式运行在独立的 Kubernetes 节点上（ GCP 的 n2d-standard-8 ，8 个 vCPU 、32 GB 内存）。SigNoz 和 ClickStack 将日志存储在 ClickHouse 中，而 ClickHouse 本身无法直接接收日志采集器发送的日志。因此每个技术栈都会在 ClickHouse 所在的节点上运行自己的 Collector ，这些 Collector 基于 OpenTelemetry Collector 构建。  我们同时向每个系统发送相同的日志。写入速率从 10k 行/秒开始，每 10 分钟增加 10k ，最高到 180k 行/秒。一个系统的最大写入速率，是指它在一个完整的 10 分钟阶梯内平均能成功存储的最高速率。如果系统发生崩溃，我们只统计它首次崩溃之前的阶梯，因为崩溃本身就是它的极限。  我们统计的是每个系统实际存储的行数，而不是它接收的行数。例如，SigNoz 和 ClickStack 在 Collector 收到日志后就会立即返回响应，因此 Collector 一旦崩溃，就会丢失已经被接收的日志。Loki 是个例外：它没有低成本的方法统计已存储的行数，但它会在返回响应之前把每一行写入 WAL （ Write-Ahead Log ，预写日志），所以对于 Loki ，我们统计的是接收的行数。 写入速率只是故事的一半。为了展示它与查询速度之间的关系，我们以 35k 行/秒的速率向 VictoriaLogs 、Elasticsearch 和 Loki 写入相同的日志，持续 4 小时，然后在冷缓存状态下对几条常见查询进行计时。我们会在后续文章中深入讨论查询。 每个系统接收的负载 日志可以是任何东西，从纯文本到包含数百个字段的 Wide Events （宽事件）。本测试选取了中间情况：每行包含 30 到 40 个字段的结构化日志，这在 Kubernetes 中很典型。其中约 20 到 30 个字段由应用写入，日志采集器再额外添加约 10 个字段，例如命名空间、节点和标签。 设想一个有 125 个节点的 Kubernetes 集群，每个节点运行 40 个属于不同应用的 Pod 。每个 Pod 的日志构成一个日志流，因此这个集群共有 5,000 个日志流。这是一个中等规模的集群，与我们在 VictoriaLogs 用户中见到的许多集群类似。  我们想看看每个系统基于自身设计能走多远。因此，每个系统都以它偏好的格式接收日志。例如，Loki 通常从 Grafana Alloy 接收 protobuf 格式。我们没有向所有系统发送同一种格式（如 OTLP ）。如果那样做，我们测量的就不仅是各系统存储日志的能力，还包括它们转换该格式的效率。 在实际场景中，日志通常以 JSON 形式输出到 stdout 。因此日志采集器需要对其进行转换，而这同样会消耗 CPU 。如果有足够多的读者感兴趣，我们也可以用一种通用格式（如 JSON 或 OTLP ）重新运行测试。 测试结果 主写入测试我们运行了 3 次，下面的结果取 3 次运行的中位数。完整的数据表格见[仓库]( github.com/VictoriaMetrics/logs-benchmark/tree/ma…)。 对于 SigNoz 和 ClickStack ，CPU 和内存包含了 Collector 、ClickHouse 及其辅助服务。 本文中提到的少数额外测试（例如查询测试）是仓库之外的一次性运行。 最大写入速率（行/秒）  - Loki 写入最多，达到 139k 行/秒。 - VictoriaLogs 位居第二，为 120k 。 - Elasticsearch 、OpenSearch 和 ClickStack 在 28k 到 48k 之间。 - SigNoz 达到 10k 行/秒，并在 20k 时崩溃。 一行日志约为 930 字节的 JSON ，因此 120k 行/秒约等于 106 MiB/s ，即每天约 9 TiB 的原始日志。 每个数字的背后都伴随着在查询速度、成本或日志类型上的权衡。我们会在[为什么会是这样的结果](#为什么会是这样的结果)中进行解释。 CPU 使用量（核） 各写入速率下的平均 CPU 核数。每条曲线终止于该系统的最大写入速率。  - 在 10k 行/秒（所有系统都能承受的唯一速率）下，VictoriaLogs 和 Loki 使用 0.4 核，ClickStack 0.9 核，Elasticsearch 1.0 核，OpenSearch 1.5 核，SigNoz 4.2 核。 - 差距随速率增长而扩大。在 40k 行/秒时，VictoriaLogs 和 Loki 使用 1.6 核，ClickStack 4.0 核，Elasticsearch 5.2 核。 - 在 120k 行/秒时，Loki 使用 6.1 核，VictoriaLogs 使用 7.2 核。 在本次测试中，CPU 是大多数系统的瓶颈。 内存使用量（ GiB ） 各写入速率下的峰值内存，不含操作系统的 Page Cache 。每条曲线终止于该系统能跟上的最后一个速率。  - VictoriaLogs 在所有速率下内存占用都最少，在 40k 行/秒时比 Loki 少 14 倍。 - Elasticsearch 和 OpenSearch 无论速率如何都使用约 17 GiB ，因为它们在启动时会预留固定大小的堆内存。默认情况下，Elasticsearch 会将堆大小设为机器内存的一半（[源码]( github.com/elastic/elasticsearch/blob/v9.5.5/dist…)）。OpenSearch 不会自动设置堆大小，所以我们给它配置了相同的大小。 - ClickStack 在 10k 行/秒时使用 1.3 GiB ，在 40k 行/秒时使用 2.2 GiB 。 - SigNoz 在 10k 行/秒时使用 3.1 GiB ，这是它在图表中唯一的数据点。 内存决定了机器需要多大、成本多高，以及还剩多少内存可用于缓存和查询。 磁盘空间 下图展示了每个系统存储一百万行日志所需的磁盘空间。作为参考，一百万行日志以原始 JSON 形式约占 887 MiB 。SigNoz 标为 N/A ，因为它的 Collector 反复崩溃，存储的行数太少，且多次运行之间的数据波动太大，无法进行比较。  - VictoriaLogs 所需磁盘空间最少，其次是 ClickStack 和 Loki。 - Elasticsearch 所需空间几乎是 VictoriaLogs 的 5 倍，OpenSearch 则是 10 倍。 日志通常会保留数周，因此磁盘空间增长很快。例如，以 50k 行/秒的速率保存 30 天的日志，VictoriaLogs 约需 7 TiB ，Loki 约需 12 TiB ，Elasticsearch 约需 33 TiB 。 磁盘写入量 下表展示了每个系统每存储一行日志向磁盘写入的字节数。这个数值会大于最终的磁盘占用，因为每个系统都会不时重写数据，以保持数据紧凑并便于搜索。N/A 表示该系统无法跟上对应的速率。 | 系统 | 10k | 20k | 40k | 80k | 120k | 150k | | --- | --- | --- | --- | --- | --- | --- | | Loki (protobuf) | 1,108 | 1,234 | 1,411 | 1,251 | 1,210 | N/A | | VictoriaLogs | 323 | 284 | 226 | 223 | 163 | N/A | | Elasticsearch | 1,393 | 1,815 | 1,917 | N/A | N/A | N/A | | OpenSearch | 2,295 | 3,197 | N/A | N/A | N/A | N/A | | SigNoz | 1,420 | N/A | N/A | N/A | N/A | N/A | | ClickStack | 414 | 461 | 572 | N/A | N/A | N/A | VictoriaLogs 每行写入磁盘的数据量同样最少，而且在更高速率下写得更少，因为一次写入更多行时压缩效果更好。Loki 每行的写入量是它的 3 到 7 倍，具体取决于速率。 这一点很重要，因为每块磁盘每秒能写入的字节数都有上限，而更快的磁盘也更贵。在我们的测试中，使用的是 GCP 上 500 GiB 的 pd-balanced 磁盘，最高写入速度为 280 MiB/s ，所以没有任何系统在耗尽 CPU 之前触及这个上限。尽管如此，VictoriaLogs 能让你使用更便宜的磁盘；在自有硬件上，它对 SSD 的磨损也要慢得多。 为什么会是这样的结果 Loki Loki 的高写入速率让我们感到意外，但它的设计可以解释这一点。Loki 的设计目标是存储大量日志，之后只在较短的时间范围内读取少数几个日志流。因此它只为流标签（这里是节点和应用）建立索引，并原样存储每一行日志。相比之下，VictoriaLogs 会把每一行拆分为字段，并分别存储每个字段，从而让后续的查询更快。在 120k 行/秒时，这会比 Loki 多消耗约 18% 的 CPU 。 向 Loki 发送日志的方式也很重要。Grafana Alloy 和 Vector 默认发送 protobuf ，而 Fluent Bit 、Fluentd 和 Logstash 发送 JSON 。在另一次单独的测试中，使用 JSON 时 Loki 只达到约 99k 行/秒，低于 VictoriaLogs ，因为处理 JSON 比处理 protobuf 消耗的 CPU 多得多。所以如果你使用 Loki ，建议优先选择 Alloy 或 Vector 。 Loki 的设计有 3 个代价。第一个代价是内存。Loki 会把每个流最新的日志行保存在内存中，以便将它们一起压缩。这意味着它的内存会随着流的数量增长。集群越大，流就越多；任何集群在部署期间流也会增多，因为每个新 Pod 都会产生一个新的流。为了观察这会占用多少内存，我们还分别用 10,000 和 15,000 个流运行了测试。  在 15,000 个流时，Loki 在 60k 行/秒时因 OOM 被杀死。而 VictoriaLogs 在流数量增加到 3 倍的情况下，仅多用了 0.2 GiB 内存。更多的内存意味着更大的机器，也意味着可用于缓存查询数据的内存更少。 第二个代价是磁盘。Loki 把日志行当作纯文本进行压缩。它不像 VictoriaLogs 和 ClickHouse 那样利用行内的字段来获得更好的压缩效果，因此它所需的磁盘空间是 VictoriaLogs 的 1.8 倍。 第三个也是最重要的代价是查询时间。Loki 只能通过标签来缩小查询范围。除此之外的任何条件，比如一个单词或一个字段值，都需要对匹配流中的每一行进行解压和扫描。在相同的日志上，前文提到的查询耗时如下。 | 查询问题 | 时间范围 | VictoriaLogs （秒） | Loki （秒） | | --- | --- | --- | --- | | 统计某个应用的错误数 | 1 小时 | 0.06 | 3.5 | | 统计每个应用的错误数 | 1 小时 | 0.59 | 171 | | 包含某个常见词的最新 100 行 | 1 小时 | 2.5 | 32 | | 每个应用的 p99 延迟 | 1 小时 | 1.9 | 302 | | 前 10 个请求路径 | 1 小时 | 8.8 | 被拒绝，超过 500 个 Series | 这些都是排查故障、构建 Dashboard 和告警时的日常问题。除第一条外，Loki 在所有查询上都要花费半分钟甚至更久，或者直接拒绝执行。第一条查询只通过标签读取一个应用的数据，即便如此，它也比 VictoriaLogs 慢了近 60 倍。 Loki 正在增加更快搜索字段的方法，例如 Bloom Filter （布隆过滤器），但这些功能仍处于实验阶段，默认关闭，并且在单节点上不可用，所以我们没有启用它们。 这些缓慢的查询并不意味着 Loki 总体上是一个糟糕的日志方案。它的设计目标是把日志低成本地保存在对象存储中，并在需要执行大型查询时横向扩展到许多查询工作节点上。在我们的单台机器上，统计一小时内每个应用的错误数花了 171 秒，大约使用了 8 核中的一半。粗略估算，若要在约 2 秒内得到结果，同样的扫描大约需要 300 个核。所以，如果你想把大量日志低成本地存储在对象存储中，并且要么只读取其中一小部分，要么愿意为大量 CPU 核付费以实现快速搜索，那么 Loki 适合你。 如果这超出了你愿意承担的成本，而你仍然需要 Dashboard 和搜索在几秒内返回结果，请继续往下读。 Elasticsearch 和 OpenSearch OpenSearch 是 Elasticsearch 的一个 Fork ，所以本节主要介绍 Elasticsearch ，最后再说明 OpenSearch 的不同之处。 Loki 几乎不建索引，并在查询时为此付出代价。Elasticsearch 则恰恰相反。它以 3 种形式存储每一行日志： - 该行的原始 JSON。 - 倒排索引，一个将每个单词和值映射到包含它们的日志行的字典，因此搜索可以直接定位到匹配的行。 - 再以列的形式存储一遍字段（ doc values ），用于排序和聚合。  这一切都让写入变得昂贵。在 40k 行/秒时，Elasticsearch 使用了 5.2 核，而 VictoriaLogs 和 Loki 只用了 1.6 核；它在约 48k 行/秒时耗尽了 CPU 。瓶颈在于 CPU ，而不是磁盘。Elasticsearch 会在返回响应前把每个请求写入磁盘，但我们在另一次单独测试中关闭了这一行为，写入速率几乎没有变化。 我们的测试使用了 [LogsDB]( elastic.co)，这是 Elasticsearch 针对日志推出的更新、更紧凑的索引模式。但即便如此，Elasticsearch 使用的磁盘空间仍是 Loki 的 2.6 倍，几乎是 VictoriaLogs 的 5 倍。 作为回报，Elasticsearch 查找数据的速度比 Loki 快得多。在 Loki 需要几分钟的查询上，Elasticsearch 只需几秒甚至更短。下面是它与 VictoriaLogs 在几类常见查询上的对比。 | 查询问题 | 时间范围 | VictoriaLogs （秒） | Elasticsearch （秒） | | --- | --- | --- | --- | | 统计包含某个常见词的行数 | 1 小时 | 6.5 | 0.10 | | 统计每个应用的错误数 | 1 小时 | 0.59 | 0.46 | | 每个应用的 p99 延迟 | 1 小时 | 1.9 | 3.9 | | 前 10 个请求路径 | 1 小时 | 8.8 | 14.7 | | 统计包含某个双词短语的行数 | 1 小时 | 4.4 | 175 | Elasticsearch 查找单个词或值几乎是瞬间完成的，在简单计数（例如统计每个应用的错误数）上也更快。而一个短语，比如 "connection timeout"，则花了它大约 3 分钟。 为了节省磁盘，它针对日志的默认配置不会存储每个单词在行中的位置，因此它必须重新读取日志行，才能判断两个词是否相邻。VictoriaLogs 在这类查询上更快，在百分位数和 Top 值查询上也更快。 Elasticsearch 之所以流行，是因为它能快速搜索任意字段。人们对它的抱怨通常集中在两点： - 成本。 为每个字段建立索引会消耗大量 CPU 、内存和磁盘。 - 运维。 运维大型集群需要投入大量精力。 Uber 同时遇到了这两个问题。索引成本高昂，而且 Uber 在每个区域运行了 20 多个集群，以确保一个集群故障不会影响全局（[Uber]( uber.com)）。Cloudflare 则不得不对部分日志进行采样，因为全部存储需要数百 TB 的空间（[Cloudflare]( blog.cloudflare.com)）。两家公司都把部分最大规模的日志负载迁移到了 ClickHouse 。 OpenSearch 的写入量略高于 Elasticsearch 的一半（ 28k 对 48k 行/秒），磁盘占用约为后者的两倍。它与 Elasticsearch 在两项默认设置上有所不同： - 它没有类似 LogsDB 的模式，因此对每行原始 JSON 的压缩没那么紧凑（[OpenSearch 文档]( docs.opensearch.org)），也不会预先排序。 - 它没有内置的日志模板，因此大多数字符串字段会被索引两次：一次按分词，一次按精确值。 OpenSearch 与 Elasticsearch 共享相同的内核，因此通过更强的压缩、排序以及完整的日志模板，你很可能可以弥补大部分差距。我们刻意让它保持接近默认配置，以展示开箱即用时的表现。 Loki 以牺牲查询速度为代价换取快速写入。Elasticsearch 以牺牲写入速度和存储为代价换取快速搜索。那么 VictoriaLogs 又提供了什么呢？ VictoriaLogs 如今，许多公司都在输出结构化日志，每一行都带有命名字段，例如日志级别、HTTP 状态码或请求耗时。VictoriaLogs 利用了这类日志的两个特性，使写入和查询都能保持高速。 1. 同一个流中的日志看起来很相似。它们的字段几乎相同，而且许多字段几乎不变。每一行的命名空间都相同，日志级别也只有少数几个取值。VictoriaLogs 会把每个流的日志存放在一起，并把每个字段存储在独立的列中，这比存储整行日志的压缩效果好得多。 2. 许多查询只需要少数几个字段。例如，统计每个应用的错误数只需要应用和级别两个字段。VictoriaLogs 只读取查询所需的列。  这种存储方式本身就让 VictoriaLogs 在磁盘上非常紧凑，聚合也很快，所以它在写入时只需构建轻量的索引和少量元数据。这就是为什么它能以 120k 行/秒的速率写入，是 Elasticsearch 的 2.5 倍，同时磁盘占用少 4.7 倍。正如前文所述，只有 Loki 的写入速度更快。更多细节请参阅 [VictoriaLogs 如何存储你的日志]( victoriametrics.com)。 在查询方面，VictoriaLogs 在几秒内响应了所有查询，并且在每一条查询上都比 Loki 快。与 Elasticsearch 相比则互有胜负，我们会在关于查询的文章中详细分析。为了保持低成本的写入和存储，VictoriaLogs 不会构建完整的倒排索引。其代价是，搜索一个常见词（出现在大量行中的词）会比较慢。而查找一个罕见词（例如用户 ID ）依然很快，因为 Bloom Filter 可以让它跳过不可能包含该词的数据。 VictoriaLogs 使用的内存也少得多。在 40k 行/秒时，Loki 使用的内存约为它的 14 倍（ 11.4 GiB 对 0.8 GiB ）；在 120k 行/秒时约为 5 倍（ 12.8 GiB 对 2.5 GiB ）。这些空闲内存并不会被浪费。操作系统会用它来缓存最近写入或读取的数据，因此许多查询（尤其是针对近期日志的查询）可以直接从内存而不是磁盘读取数据。它也为大量并发查询留出了空间，我们计划在后续关于查询的文章中对此进行测试。 公平地说，本次基准测试使用的是结构化日志，这正是 VictoriaLogs 的优势所在。而针对这类日志的日常工作，很大一部分也是用于 Dashboard 、告警和分析的聚合，这正是列式存储最能发挥作用的地方。如果你的日志是非结构化的，例如老旧应用和网络设备输出的纯文本日志，VictoriaLogs 依然能很好地写入和搜索它们，但收益会比结构化日志小一些。 SigNoz 和 ClickStack SigNoz 和 ClickStack 比 Loki 更让我们意外。两者都把日志存储在 ClickHouse 中，而 ClickHouse 和 VictoriaLogs 一样是列式数据库，所以我们原本预期它们的表现会接近 VictoriaLogs 。但事实并非如此。ClickStack 达到 46k 行/秒，SigNoz 只有 10k ，而 VictoriaLogs 为 120k 。在磁盘空间方面，ClickStack 排名第二，但每百万行仍需要 82 MiB ，是 VictoriaLogs 的 1.5 倍。 较低的写入速率来自两个搭建在 ClickHouse 之上的方案，来自于它们的 Collector 。ClickHouse 是一个通用数据库，无法直接接收常见格式的日志，因此需要由一个 OpenTelemetry Collector 先把日志转换成行数据。这个 Collector 消耗的 CPU 比 ClickHouse 本身还多。在 SigNoz 中，10k 行/秒时 Collector 使用了 2.7 核，而 ClickHouse 为 1.5 核；在 ClickStack 中，达到最大速率时两者分别为 3.3 核和 2.2 核。  更大的磁盘占用可能源于两个技术栈存储字段的方式。它们把日志存储在一张固定列的 ClickHouse 表中，但 [OpenTelemetry 日志格式]( opentelemetry.io) 只有少数几个固定字段，外加可以容纳任意自定义数据的 Attributes 。固定字段（如 Timestamp 、SeverityText 和 TraceId ）拥有独立的列，压缩效果很好。而 Attributes 则被放入少数几个 Map 列中，许多字段的值混杂在一起，因此压缩效果较差，查询也更慢。 VictoriaLogs 受到了 ClickHouse 的启发，但它是专门为日志而构建的（[FAQ]( docs.victoriametrics.com)）。它与 ClickHouse 有两点不同，这很可能解释了写入速率和磁盘空间上的大部分差距： - 无需 Collector 。 VictoriaLogs 可以直接接收日志采集器已经在发送的各种格式，例如 OpenTelemetry 、Loki 、Elasticsearch 、Datadog 和 Splunk 。 - 无需 Schema 。 日志行中的每个字段都有自己独立的列，而不仅仅是少数几个固定字段。 在本次测试中，基于相同设计的 SigNoz 使用的 CPU 几乎是 ClickStack 的 5 倍，主要消耗在其 Collector 上。Collector 把这些 CPU 花在了额外的元数据上，这些元数据可以加速查询并为其 UI 提供支持，我们会在关于查询的文章中探讨这一权衡。此外，它默认不设内存上限，而我们保留了这一默认配置。因此当它处理不过来时，会把日志行一直堆积在内存中，直到被 Kubernetes OOM Kill 。其他用户也报告过类似的 Collector 崩溃问题（[GitHub issue]( signoz#6128)）。 公平地说，SigNoz 和 ClickStack 是完整的可观测性技术栈，它们把日志、指标和链路追踪汇集在一处，并把每条链路关联到对应的日志。本次测试只衡量了其中的一部分：日志写入。两个技术栈也都建议随着数据量增长增加更多的 Collector 。ClickHouse 的文档按约 3 核处理 60k 事件/秒来规划单个 Collector （[ClickHouse 文档]( clickhouse.com)），而 SigNoz 自己的基准测试在一台 16 核机器上，于 ClickHouse 旁运行了 3 个 Collector ，达到约 55k 行/秒（[SigNoz]( signoz.io)）。在我们的测试中，为了对所有系统公平，每个系统都只分配了一台 8 核机器，所以 Collector 不得不与 ClickHouse 共享资源。集群测试中各个组件可以独立扩展，能够展示它们从更大规模的部署中能获得多少收益，这也在我们的计划之中。 关键结论 每个系统都根据自身的优先级和架构做出了不同的权衡。 - Loki 写入最多，因为它建立的索引最少。代价是查询缓慢，除非你为更多 CPU 核付费来加速查询。 - Elasticsearch 和 OpenSearch 为每一行的每一个词建立索引。这让它们能够非常快速地查找单词和精确值。但代价是写入时 CPU 消耗高，并且需要大量存储空间。 - SigNoz 和 ClickStack 把日志存储在 ClickHouse 中。ClickHouse 并非专为日志而构建，因此两个技术栈都需要额外的组件，这会拖慢写入速度并增加成本。如果你希望把 OTLP 日志、指标和链路追踪集中在一处，它们会比较适合你。 - VictoriaLogs 受到 ClickHouse 的启发，但它是直接为日志而构建的。它充分利用现代结构化日志的特点，实现快速写入、秒级查询响应，并且在内存和磁盘上的成本最低。如果你的日志是纯文本，收益会小一些。 本文在单台机器、普通云盘上测试了结构化日志。在后续文章中，我们计划深入介绍查询、集群模式，以及在 VictoriaLogs [支持对象存储]( VictoriaLogs#1155)后的对象存储测试。 你可以使用我们的[基准测试仓库]( VictoriaMetrics/logs-benchmark)复现主要的写入测试结果，也可以通过[快速入门指南]( docs.victoriametrics.com)试用 VictoriaLogs 。如果你发现我们的测试环境有错误，或者知道如何更好地调优某个系统，欢迎在[仓库]( github.com/VictoriaMetrics/logs-benchmark/issues)中提交 issue ，或在 [VictoriaMetrics 的 Slack]( slack.victoriametrics.com) 上联系我们。

Empty AI observability panels: a product teardown of SigNoz I’ve been building ReplayLens, a project around AI agent quality and regression reporting. That got me interested in an engineering question: how do we make complex technical systems easier to understand when something isn't working as expected? While reading SigNozs public AI observability documentation I noticed a product-design question. When a developer opens an observability dashboard and sees panels how quickly can they figure out why? This is a documentation-based teardown. I have not reproduced a bug in a running SigNoz instance so I'm treating this as a hypothesis to investigate than a confirmed defect. 1. The details SigNozs AI Observability features help developers inspect LLM and agent workloads, including token usage, cost, latency, errors time to first token and tool calls. The documentation explains that these panels depend on telemetry arriving in the expected format. For example: - gen_ai.request.model is used by LLM panels and the model filter. - gen_ai.usage.input_tokens and gen_ai.usage.output_tokens support token and cost panels. - gen_ai.tool.name is used by tool-call panels. - AI Observability requires traces; integrations that send metrics and logs do not appear in these views. SigNoz also provides Attribute Mapping to normalize attributes emitted under names by instrumentation libraries. Sources - AI Observability overview - Telemetry requirements - Attribute Mapping These are engineering requirements. Standardized attributes make telemetry more consistent across frameworks and providers. The question I want to explore is how directly the product helps a developer identify which requirement is missing. 2. The potential friction Imagine instrumenting a LLM application and opening AI Observability to understand its token consumption. The panels are empty. Possible explanations include: 1. The application isn't sending traces. 2. Traces arrive,. The expected GenAI attributes aren't present. 3. The instrumentation uses attribute names. 4. Model pricing isn't configured, so cost calculations aren't available as expected. 5. A self-hosted installation is missing a required traces-pipeline processor. Each situation calls for a fix. The documentation describes the prerequisites and configuration options. My product hypothesis is that developers would benefit from seeing more of this information directly in the workflow. An empty panel shouldn't force a developer to guess which layer is responsible. I have not verified how every one of these cases currently appears in the product. That would require a test. 3. Three improvements I'd explore A. A telemetry-readiness checklist A small diagnostic panel could report whether the system has detected: - Incoming traces - LLM spans - The model attribute - Input and output token attributes - Model pricing configuration - Required processors for self-hosted setups Each status could link to the appropriate setup instructions. The goal would be to reduce the time between seeing a panel and identifying the next useful debugging step. B. Informative empty states Instead of a generic empty chart a panel could explain the requirement it needs. For example: No token usage data found “We haven't found matching spans with the attributes this panel uses. Inspect a span. Review your attribute mapping.” The exact message should depend on what the system has checked. It shouldn't tell someone an attribute is missing if that hasn't been verified. C. A visible validation path SigNozs Attribute Mapping documentation already describes testing mappings against a sample span and inspecting the resulting attributes. I'd explore bringing that validation flow to the moment a developer encounters missing data: 1. Select a span. 2. Highlight expected attributes that're present or absent. 3. Preview the mapped result. 4. Explain which panels the resulting attributes support. 5. Send a span and verify the result. The idea isn't to replace the existing configuration. It's to make the connection between configuration and outcome easier to understand. 4. How I'd validate the hypothesis I'd test this against a running instance before filing an issue or calling it a bug. I'd send a set of controlled examples: - A trace containing the expected GenAI attributes. - A trace missing token-usage attributes. - A trace using attribute names. - An application sending metrics and logs but no traces. - A self-hosted setup with the required processors configured correctly and incorrectly. For each case I'd record the UI state, available diagnostics steps needed to resolve the problem and whether a developer could identify the cause without repeatedly consulting documentation. That would establish whether this is an usability gap, an already well-handled case or an opportunity for further improvement. 5. What I like about the existing approach I wouldn't change the underlying commitment to OpenTelemetry conventions. Standard attribute names make it easier to compare telemetry across applications and attribute mapping addresses the reality that different instrumentation libraries can emit names. The documentation also makes important prerequisites explicit. I'd build on those strengths by making the diagnostic experience as clear as the setup guidance. The takeaway Developer tools need to do more than display the data when everything is configured correctly. They should also help developers understand why the expected data isn't there. For AI observability making the gap between instrumentation and useful dashboards easier to diagnose is a product question. I'd be interested, in how the SigNoz team thinks about telemetry readiness and empty-state diagnostics. References - AI Observability - Telemetry requirements - Attribute Mapping - SigNoz source code --- This is a documentation-based product teardown. No live benchmark or confirmed bug reproduction is claimed.

An API slowed down. Where do you look first? 📈 SigNoz brings traces, logs and metrics into one observability dashboard. Follow the clues behind an incident. 🔎 SigNoz/signoz #Observability #浇朋友 #蓝朋友 #互关 #fo

接口一变慢，日志和指标得翻好几个平台？📈 SigNoz 集中查看调用链、日志和指标，方便定位服务异常。开发和运维都能用。🔎 SigNoz/signoz #Observability #浇朋友 #蓝朋友 #互关 #fo

I Built Scrapers That Heal Themselves (and Won an NVIDIA DGX Spark) Scraper Factory took first place at the WeMakeDevs Zero Downtime Hackathon in San Francisco. On the surface it is a small product. It scrapes about twelve days of TV guide data for Philo, YouTube TV and Sling, ranks the top ten upcoming shows and the top ten upcoming movies, and gives you one click to record each title on Philo. The interesting part is underneath. I wanted scrapers that notice when they break, attempt their own repair, and only accept that repair once it is proven correct. Three tools carried the project, each doing what it is best at: Bright Data for scraping and self-healing, Port as the control plane, and SigNoz for the audit trail. This post walks through how they fit together, with the code and config that make each piece work, so you can follow along or run it yourself. Try the product: scraper-factory-top20.vercel.app Source code: Nitishmane/scraper-factory !The product: the top 10 shows and top 10 movies, with a one-click Record on Philo button where a direct link exists. The product: the top 10 shows and top 10 movies, with a one-click Record on Philo button where a direct link exists. The problem: scrapers rot without anyone noticing Every scraper I have built eventually fails. A site ships a redesign, a selector stops matching, and the data either stops arriving or, worse, keeps arriving slightly wrong. Times land in the wrong column, titles swap with descriptions, rows go missing. You usually find out a week later when something downstream looks odd. Most teams treat scrapers as scripts. Scripts don't have owners, health states or an on-call rotation. I wanted each scraper treated like any other production service: state you can see, automation that reacts to that state, and a clear record of every repair. The one rule that ties the project together: judgment goes to the agents, proof stays deterministic. A model decides what broke and how to repair it. Plain code, diffing against a frozen fixture, decides whether the repair is real. How the pieces fit Here is the production data path. Port triggers the scrape, Bright Data extracts the guide, the pipeline ranks titles with TMDB data, and the results land back in Port's Context Lake, where the webapp reads them. !The data pipeline, from Port's scheduled trigger to the published top 10 lists. The data pipeline, from Port's scheduled trigger to the published top 10 lists. In production this runs weekly without a laptop. Port's wf_scheduled_scrape workflow fires every Monday at 08:00 UTC and dispatches a GitHub Actions job that runs scrape, heal (if needed), catalog refresh, rank and publish, then exits. The same job can be started on demand from a Port button. Bright Data: scrapers that repair themselves The usual approach to scraping is structural: you find that the airtime lives in div.card > span:nth-child(3) and write that down. That selector says where the data sits, not what it means. Rename a CSS class and it breaks. Reorder the cards and it breaks. Bright Data's Scraper Studio builds a collector from a natural language description of the data instead of selectors. A description of meaning survives a redesign far better than a selector does. And the prompt becomes the thing you heal from: when a scraper drifts, the scraper heal command reasons over that same description plus a plain language report of what went wrong. A prompt about meaning This is the full prompt behind the channel schedule collector, from src/factory/pipeline.py: PROMPT = ( "This page lists a TV channel's upcoming schedule as program cards. Extract every " "program card. For each: the program title as title, the episode title if shown as " "episode_title, the airing date exactly as displayed (like Sat, Aug 22) as date_raw, " "the start time exactly as displayed (like 2:00 PM) as start_raw, the time range text " "if shown (like Sat, Aug 22 2:00 PM - 2:30 PM) as time_range_raw, the category text " "as category. Include every card." ) There are no class names, positions or DOM paths in it. Notice also date_raw and start_raw. I always capture the string exactly as displayed next to any parsed value. If start_raw says "8:00 PM" but the parsed UTC time is null, that is a drift signal, and one of the detection rules checks for exactly that. Everything goes through the Bright Data CLI, which made it easy to script from Python. Creating a collector is one command: brightdata login --api-key $BRIGHTDATA_API_KEY brightdata scraper create "$FIXTURE_BASE_URL/channel.html" "<PROMPT>" brightdata scraper run <collector_id> --urls "<channel page url>" --pretty One collector, three providers The guide data comes from a third party site where every channel has its own page: roughly 455 program cards covering about twelve days. Because every channel page has the same structure, one shared collector serves all three providers (137 Philo channels, 175 YouTube TV channels and 119 Sling channels). A second collector reads Philo's browse pages, about 739 shows and movies, so the ranked list can link straight to a record button. One practical tip: generate the collector against a small, clean sample. I trimmed a real channel page down to 30 program cards, removed links the generator might try to follow, served it through a tunnel, and created the collector from that. That same trimmed page later became my frozen test fixture. Splitting fetch and extract Bright Data's usage policy does not allow collector runs directly against streaming media domains. After checking with their support team, I split the job across two Bright Data products. The Web Unlocker fetches the live page and archives it. The Scraper Studio collector then extracts from that archived copy, served through a tunnel. !Fetch with the Web Unlocker, extract with a Scraper Studio collector. Fetch with the Web Unlocker, extract with a Scraper Studio collector. The code still tries a direct collector run first, so if the policy ever changes the relay step drops away on its own. A side benefit: every scrape's input HTML is archived, so any drift investigation can be replayed byte for byte. !The two Scraper Studio collectors: channel-schedule-v3 for every guide page, and philo-catalog for record links. The two Scraper Studio collectors: channel-schedule-v3 for every guide page, and philo-catalog for record links. The repair loop When drift is detected, the healer calls scraper heal with the collector ID and a plain language symptom description. Bright Data returns a proposal in an awaiting_approval state, with a preview_result of what the repaired scraper would extract. Nothing changes until my code decides. Here is the core of src/factory/agents/healer.py, trimmed for length: proposal = brightdata.heal(collector_id, drift_description) # CLI: brightdata scraper heal <collector_id> "<symptom>" # -> {"status": "awaiting_approval", "preview_result": [...]} verdict = verifier.verify("channel", proposal.preview_result) # no model if verdict.passed: brightdata.approve(collector_id) # scraper approve <id> port.set_scraper_health(collector_id, "healthy", verify_status="pass") else: brightdata.approve(collector_id, reject=True) # approve <id> --reject port.set_scraper_health(collector_id, "broken", verify_status="fail") # heal_event recorded as rolled_back; a person takes it from here The awaiting_approval state is what makes this safe. The model proposes, my code checks, and only then does anything change. Because every run goes through the CLI, the Bright Data control panel doubles as a run log. Each weekly scrape shows up as a batch of collector runs with inputs, records and success rate: !Run history for the channel schedule collector. The Monday batches are the weekly production scrape. Run history for the channel schedule collector. The Monday batches are the weekly production scrape. The gate: detection and verification in plain code Two parts of the system never involve a model: deciding that something is wrong, and deciding that a fix is right. Seven drift rules Every run passes through seven deterministic checks in src/factory/detect.py: rows per page out of band, required field null rate, raw time present but parsed time null, channel collapse, empty page rate, a change in the schema hash of returned keys, and lineup disappearance. The thresholds are plain constants: REQUIRED_FIELDS = ("channel", "title", "start_utc") NULL_RATE_LIMIT = 0.10 EMPTY_PAGE_LIMIT = 0.20 # >20% of channel pages returning nothing is breakage CHANNEL_COLLAPSE = 0.80 # <80% of requested channels yielding rows is breakage LINEUP_DELTA_LIMIT = 0.10 # >10% channel-set churn needs two confirming runs PAGE_ROW_BANDS = (15, 800) A broken scraper and a real change at the source look identical in one sample. If 15 percent of channels vanish, that could be breakage or a real carriage change. So lineup churn needs two consecutive runs agreeing before it is accepted, and drift state is keyed by collector:provider so one provider's lineup is never compared against another's. The verifier The verifier normalises the heal preview and the hand-checked expected output for the frozen fixture, then diffs them. It is the only thing allowed to promote a repair: COMPARE_FIELDS = ("title", "date_raw", "start_raw") def verify(provider: str, rows: list[dict]) -> Verdict: want, got = _normalise(expected(provider)), _normalise(rows) if want == got: return Verdict(True, []) diff = [] missing = [w for w in want if w not in got] extra = [g for g in got if g not in want] if len(want) != len(got): diff.append(f"row count {len(got)}, expected {len(want)}") for item in missing[:3]: diff.append(f"missing {item}") for item in extra[:3]: diff.append(f"unexpected {item}") return Verdict(False, diff) The fixture is three files: a real channel page trimmed to 30 cards, its hand-checked expected output, and a mutated copy with every structural handle moved (classes renamed, cards reordered, airtimes wrapped in new elements). The mutated copy lets me inject drift on demand with make demo-drift. !The self-heal loop: three ways to detect a problem, one repair path, and a deterministic gate before anything is promoted. The self-heal loop: three ways to detect a problem, one repair path, and a deterministic gate before anything is promoted. Port: the control plane Scrapers needed to be entities with state, not scripts. Port's data model made that possible. Seven blueprints model the system: data_source, scraper, scrape_run, heal_event, ranked_title, requirement and factory_deployment. Each scraper has a health property that moves between healthy, drifting and broken. Every run is a scrape_run entity. Every repair attempt is a heal_event recording its trigger, the drift description, the outcome and a link to the trace. !Both scrapers as Port entities, with collector ID, verify status and health. Both scrapers as Port entities, with collector ID, verify status and health. Automations that react to state The fastest repair path is a Port automation on the scraper blueprint. A JQ condition on the updated entity means it only fires when health flips to drifting: { "identifier": "invoke_healer_on_drift", "trigger": { "type": "automation", "event": { "type": "ENTITY_UPDATED", "blueprintIdentifier": "scraper" }, "condition": { "type": "JQ", "expressions": [".diff.after.properties.health == \"drifting\""], "combinator": "and" } }, "invocationMethod": { "type": "WEBHOOK", "url": "<factory>/heal?token=..." }, "publish": true } Workflows with an AI agent in the middle When a run fails outright, the wf_heal_on_bad_run workflow takes over. It is triggered when a scrape_run is created with status error or with drift detected. The middle node is Port's Triage agent, which reads the scraper's run and heal history and writes a plain language symptom report. The last node passes that report to the heal endpoint: trigger -> EVENT_TRIGGER on scrape_run created (.status == "error") or (.drift_detected == true) triage -> AI_AGENT "triage" "Describe the observed SYMPTOM (what changed in the data), never a guess at the fix. Under 400 characters." heal -> WEBHOOK POST /heal { provider, collector_id, drift_description: {{ .triage.output }} } !The wf_heal_on_bad_run workflow in Port: the trigger, the Triage agent, then the call to the healer. The wf_heal_on_bad_run workflow in Port: the trigger, the Triage agent, then the call to the healer. The agent's job stays narrow on purpose. Triage writes the symptom report but never approves its own fix. That decision stays with the verifier. The other workflows are simpler. wf_scheduled_scrape runs on a cron (0 8 * * 1) and dispatches the GitHub Actions job. wf_publish_on_success publishes the rankings whenever a clean run lands. Three self-service actions, run_scrape_now, refresh_catalog_now and publish_now, let an operator force a cycle from the Port UI. Agents, scorecards and the Context Lake - Four AI agents. Triage (approval required), Heal Explainer (explains why a repair was approved or rejected), On-Call Assistant (answers operator questions) and Catalog Coverage (reports how many ranked titles have a direct record link). - A reliability scorecard. scraper_reliability grades each scraper Basic, Bronze (verified today), Silver (verification passing) or Gold (healthy and verified). - The Context Lake as the data layer. The 20 ranked_title entities are the product. The Next.js webapp has no database of its own; it reads them straight from Port. Publishing is just an upsert. !The Factory Operator dashboard in Port: scrapers by health, the architecture note and recent heal events, including the rolled back repairs. The Factory Operator dashboard in Port: scrapers by health, the architecture note and recent heal events, including the rolled back repairs. Everything above is provisioned through Port's API by one idempotent script, scripts/bootstrap_port.py, so the whole setup can be rebuilt at any time. SigNoz: the audit trail Any system that changes itself needs a clear record of what it did and why. SigNoz carried every trace, metric and log, and its alerting acted as the safety net for the repair loop. It runs self-hosted with Docker Compose, so there was no API key or trial to manage during the hackathon. Auto-instrumentation first The FastAPI service runs under OpenTelemetry auto-instrumentation and exports OTLP to SigNoz: git clone -b main https://github.com/SigNoz/signoz.git ../signoz docker compose -f ../signoz/deploy/docker/docker-compose.yaml up -d # the factory service (this is what `make serve` runs) OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ opentelemetry-instrument python -m uvicorn factory.app:app --host 0.0.0.0 --port 8000 Because the web layer is instrumented, even Port's automation calling /heal shows up as a span. On top of that I added custom spans for each pipeline step, with the attributes I would want during an incident: with tracer().start_as_current_span("scrape.heal.reprompt") as span: span.set_attribute("target.provider", provider) span.set_attribute("scraper.id", collector_id) span.set_attribute("heal.preview_rows", len(proposal.preview_result)) log().info("heal requested for %s: %s", provider, drift_description) ... span.set_attribute("verify.status", "pass" if verdict.passed else "fail") A scrape produces one scrape.run trace with a scrape.fetch span per channel page (tagged with provider, channel ID, row count and fetch mode), then scrape.extract, scrape.validate, and when a repair runs, scrape.heal.reprompt and scrape.verify.fixture. Logs carry trace and span IDs, and the drift description, heal prompt and verifier diff are all logged inside the relevant span. Two detection paths, one endpoint The inline detector marks the scraper as drifting in Port within seconds. SigNoz is the safety net: an alert rule on scraper.drift.detected > 0 over a five minute window, evaluated every minute, calls the same /heal endpoint through a webhook channel. Whichever arrives first does the work; the second finds the scraper already healthy and exits. A second rule fires when data stops arriving altogether, which is the only signal that catches a pipeline that died before it could run its own detector. Pulling in what isn't pushed Port's action and workflow runs and Bright Data's remaining budget are not emitted as telemetry by default. A small background watcher polls both and re-emits them as metrics (port.action.runs, port.workflow.runs and brightdata.budget.remaining), de-duplicated in SQLite so a restart never double counts. The dashboard, the alert and the webhook channel are all created by scripts/bootstrap_signoz.py, and the dashboard is embedded in Port's operator view. !The three correlated signals sent to SigNoz, plus the dashboard and alert. The three correlated signals sent to SigNoz, plus the dashboard and alert. The unhappy path, on purpose The clearest proof came from a fault I injected during the hackathon. A scrape run landed with an error, the heal workflow fired, the Triage agent wrote up the symptom, and the repair endpoint ran a real cycle against Bright Data. The proposal came back with a preview of 2 rows where the fixture expected 30. My verifier rejected it. Port recorded the heal event as rolled back, set the scraper to broken, and left it for a person. Nothing bad reached the product. The next clean scheduled run moved health back to healthy on its own. Opening the SigNoz trace told the whole story: the fetch spans with their row counts, the heal span holding the symptom description, and the verify span whose log read "row count 2, expected 30". No guessing, no digging through separate log files. If I had let a model approve its own repair, that two row extraction would have been promoted and the product would have shown a nearly empty guide with no alarm going off. Run it yourself You need Python 3.11+, Node 20+, Docker, a Bright Data API key, a Port account and a SigNoz instance. A free TMDB key improves the ranking. git clone https://github.com/Nitishmane/scraper-factory && cd scraper-factory cp .env.example .env # fill in credentials make setup # dependencies + OpenTelemetry instrumentation make bootstrap # Port: blueprints, workflows, agents python3 scripts/bootstrap_signoz.py # SigNoz: dashboard, drift alert, webhook channel make serve # factory API + demo site on :8000 make run && make catalog && make rank && make publish make demo-drift # inject drift and watch the repair loop make verify # the fixture gate; non-zero exit on mismatch Bright Data's collectors run in the cloud and can't reach localhost, so the fixtures and the factory's webhooks are exposed through cloudflared tunnels during development. The README covers the full setup. What I would pass on - Write prompts about meaning, not layout. The prompt is what the healer works from. - Capture the raw string next to every parsed value. Raw present but parsed null turned out to be one of my most reliable drift signals. - Treat a heal preview as a candidate. Check it against known good output before approving it. - Model the unusual things. Treating scrapers, runs and repairs as entities is what made every other feature useful. - Keep the agent's job narrow. Triage writes the symptom report; it does not approve its own fix. - Make the repair endpoint idempotent. With more than one trigger path, they will race. - Alert on a metric, not a log line. And add an alert for silence. - Keep the setup in scripts. Re-running one idempotent bootstrap beats clicking through settings at 2 AM during a hackathon. About the project Scraper Factory was built for the WeMakeDevs Zero Downtime Hackathon, where it won first place and an NVIDIA DGX Spark. Thanks to Kunal Kushwaha and the WeMakeDevs team for putting on a great event, and to Port, Bright Data and SigNoz for the tools that made it possible. Live webapp: scraper-factory-top20.vercel.app Source code: Nitishmane/scraper-factory {% github Nitishmane/scraper-factory %}

GIÁM SÁT TOÀN DIỆN APM, METRICS VÀ LOGS THAY THẾ DATADOG: SIGNOZ ỨNG DỤNG NGUYÊN BẢN CHUẨN OPENTELEMETRY Sử dụng các nền tảng giám sát ứng dụng thương mại (như DataDog hay New Relic) thường mang lại trải nghiệm tuyệt vời cho đến khi hóa đơn cuối tháng ập đến: chi phí tính theo số lượng host và gigabyte log ngốn một khoản ngân sách khổng lồ của công ty. Hơn nữa, việc bị khóa chặt vào các agent độc quyền khiến việc chuyển đổi sau này rất tốn công sức. SigNoz là nền tảng giám sát hiệu năng ứng dụng (Application Performance Monitoring - APM) và khả năng quan sát (Observability) mã nguồn mở, được thiết kế theo chuẩn OpenTelemetry nguyên bản. Tech Stack: Được xây dựng bằng Golang và React, sử dụng cơ sở dữ liệu ClickHouse hiệu năng cực cao để lưu trữ và truy vấn khối lượng lớn traces, metrics và logs. Tích hợp trực tiếp với OpenTelemetry Collector. Tại sao nên dùng? - Chuẩn OpenTelemetry 100% không lo Vendor Lock-in: Sử dụng các thư viện instrument chuẩn của cộng đồng OpenTelemetry (hỗ trợ Java, Python, Node.js, Go, .NET), bạn có thể chuyển đổi backend bất cứ lúc nào mà không cần sửa code ứng dụng. - Tương quan thông minh giữa Traces, Metrics và Logs: Khi phát hiện một API có độ trễ bất thường (P99 latency tăng vọt), bạn có thể nhấp trực tiếp từ biểu đồ sang xem chi tiết trace và đọc đúng dòng log gây lỗi của request đó. - Tiết kiệm chi phí phần cứng vượt trội: Nhờ lưu trữ trên ClickHouse, SigNoz nén dữ liệu cực tốt và xử lý các câu truy vấn phức tạp nhanh hơn nhiều so với cụm Elasticsearch truyền thống. Dự án đạt hơn 19.000 stars trên GitHub và đang là giải pháp thay thế mã nguồn mở hàng đầu cho DataDog trong cộng đồng Cloud Native. Link tham khảo: SigNoz/signoz #OpenSourceVN #VNOSS #opensourcevn #vnoss #DevCommunity #SigNoz #OpenTelemetry #APM #Observability #ClickHouse #Golang