OpenTelemetry for Beginners: Logs, Metrics and Traces in One Setup
Picture this. A customer says checkout feels slow. You open the logs and find thousands of lines.
Which request was theirs? Which step was slow? Was it just them, or everyone?
Logs alone can't answer all of that. You need three kinds of data about the same request, and a way to connect them.
That's what OpenTelemetry gives you. It has a reputation for being complicated, but the core idea is small.
In this post you'll send traces, metrics and logs from a small FastAPI app into Grafana. The backend is one Docker command. The boilerplate is handled by zero-code instrumentation. You write a few lines yourself for the interesting parts.
> Heads up on Python logs: In OpenTelemetry Python, traces and metrics are marked Stable, but logs are still marked Development. Logging works fine for this walkthrough. Just expect the logs side to change more than the other two.
What OpenTelemetry is (and what it isn't)
OpenTelemetry, or OTel for short, is an open standard and a set of tools for producing telemetry. That's the traces, metrics and logs your software emits about itself.
It's a Cloud Native Computing Foundation project, and it grew out of two older projects, OpenTracing and OpenCensus.
The key word is _producing_. OpenTelemetry doesn't store or display anything. You still need a backend for that.
The win is that you instrument once, and your code doesn't care which backend receives the data.
Here are the pieces you'll meet:
| Piece | What it does | In this post |
| --- | --- | --- |
| API and SDK | The code you call, and the engine behind it that batches and exports data | Installed with opentelemetry-distro |
| Instrumentation | Libraries that create telemetry for you (web frameworks, HTTP clients, databases) | FastAPI and logging |
| OTLP | The format and protocol the data travels in | gRPC on port 4317 |
| Collector | A service that receives, processes and forwards telemetry | Built into the Grafana image |
| Backend | Where the data is stored and viewed | Tempo, Prometheus, Loki and Grafana |
The three signals in plain words
| Signal | Plain-English meaning | Question it answers | Example |
| --- | --- | --- | --- |
| Traces | The story of one request as it moves through your code | Where did this request spend its time? | POST /orders took 380 ms, and 340 of that was charge_card |
| Metrics | Numbers measured over time | How often? How slow? Is it getting worse? | Declined orders per minute |
| Logs | Timestamped messages, in words | What exactly happened? | Payment declined: keyboard |
Two terms you'll see everywhere. A span is one step inside a trace, like "charge the card". A trace is a tree of spans.
Spans carry attributes, which are key-value details such as order.item = keyboard.
Which signal do you reach for first? It depends on the question:
!Mermaid Diagram
The real power is in the links. A metric spike leads you to a trace. A span leads you to the logs written during it.
That last jump works because each log record carries the trace ID and span ID of the request that wrote it. OpenTelemetry adds them for you.
What we'll build
A tiny orders API with one endpoint, POST /orders. It "charges a card", which is just a short sleep with a 25% chance of being declined.
That gives you slow requests and failures to look at. Here's how the data flows:
!Mermaid Diagram
Each request will produce a trace shaped like this:
!Mermaid Diagram
The official docs make a useful point here. Automatic instrumentation captures what happens at the edges of your system, like inbound and outbound HTTP requests. It can't see what's going on inside your own code.
So the top span is free. The two inner spans, the counter and the histogram are yours.
You'll need Python 3.10 or newer (that's what OpenTelemetry Python supports), Docker, and a terminal.
Step 1: Start the backend
docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm
This is Grafana's otel-lgtm image. It bundles an OpenTelemetry Collector, Prometheus for metrics, Tempo for traces, Loki for logs and Grafana to look at all three.
The ports:
- 3000 is the Grafana UI
- 4317 is where OTLP over gRPC arrives
- 4318 is where OTLP over HTTP arrives
Open localhost and log in with user admin and password admin.
> Development only: Grafana says this image is intended for development, demo and testing. The data lives inside the container, so it's gone when the container is. Don't use it as your production backend.
Step 2: Set up the project
mkdir orders-api && cd orders-api
python3 -m venv venv
source venv/bin/activate
pip install fastapi "uvicorn[standard]" opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-logging
opentelemetry-bootstrap -a install
On Windows PowerShell, activate with .\venv\Scripts\Activate.ps1 and use python instead of python3.
What each piece does:
- opentelemetry-distro brings the API and SDK, plus two commands: opentelemetry-bootstrap and opentelemetry-instrument.
- opentelemetry-exporter-otlp lets the app send data over OTLP.
- opentelemetry-instrumentation-logging connects Python's logging module to OpenTelemetry.
- opentelemetry-bootstrap -a install looks at what's installed and adds the matching instrumentation. Here, that means FastAPI.
Run bootstrap _after_ you install FastAPI, because it decides based on what it finds.
Step 3: Write the app
Create main.py:
import asyncio
import logging
import random
import time
from fastapi import FastAPI, HTTPException
from opentelemetry import metrics, trace
from pydantic import BaseModel, Field
app = FastAPI(title="orders-api")
# Let INFO records through.
logger = logging.getLogger("orders")
logger.setLevel(logging.INFO)
tracer = trace.get_tracer("orders.tracer")
meter = metrics.get_meter("orders.meter")
orders_processed = meter.create_counter(
"orders.processed",
description="Orders processed, by outcome",
)
payment_duration = meter.create_histogram(
"payment.duration",
unit="ms",
description="Time spent charging the card",
)
class Order(BaseModel):
item: str
quantity: int = Field(ge=1, le=10)
class PaymentDeclined(Exception):
"""Raised when the pretend payment provider says no."""
async def charge_card(amount: float) -> None:
started = time.perf_counter()
with tracer.start_as_current_span("charge_card") as span:
span.set_attribute("payment.amount", amount)
try:
await asyncio.sleep(random.uniform(0.05, 0.4)) # pretend to call a payment API
if random.random() < 0.25:
raise PaymentDeclined("card declined")
finally:
payment_duration.record((time.perf_counter() - started) * 1000)
@app.post("/orders")
async def create_order(order: Order):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.item", order.item)
span.set_attribute("order.quantity", order.quantity)
logger.info("Processing order: %s x%d", order.item, order.quantity)
try:
await charge_card(round(order.quantity * 9.99, 2))
outcome = "paid"
logger.info("Order paid: %s", order.item)
except PaymentDeclined:
outcome = "declined"
logger.warning("Payment declined: %s", order.item)
span.set_attribute("order.outcome", outcome)
orders_processed.add(1, {"outcome": outcome})
if outcome == "declined":
raise HTTPException(status_code=402, detail="Payment declined")
return {"status": "paid", "item": order.item}
You don't need to memorise any of it. These are the parts worth understanding.
Spans
tracer.start_as_current_span(...) opens a span for the length of the with block. A span opened inside another one becomes its child. That's how charge_card ends up under process_order.
set_attribute attaches details. Put anything here that will help you debug later, like the item or the amount.
There's a nice default too. If an exception escapes a with span block, the SDK records it on the span and marks the span as an error.
That's why a declined payment will show up as a red charge_card span, with no extra code from you.
Metrics
Two instruments, two jobs:
- Counter: counts things, and only goes up. We add 1 for every order, labelled with its outcome.
- Histogram: records measurements like durations. It lets you ask for percentiles later, such as "how slow are the slowest 5%?"
I record the duration in milliseconds on purpose. The default histogram buckets are tuned for millisecond-sized values: 5, 10, 25, 50 and so on up to 10,000.
Record seconds instead and almost everything lands in the first bucket, so your percentiles turn to mush.
> Careful with metric attributes: Every unique combination of attribute values creates a new time series. outcome has two values, which is fine. A user ID or order ID would create millions. Keep high-detail values like order.item on spans, where they're cheap.
Logs
There's no special OpenTelemetry logging call here. You keep using Python's logging, and OpenTelemetry attaches a handler that ships the records.
One gotcha: Python only lets WARNING and above through by default. So we set our logger to INFO, or the "Processing order" lines would silently vanish.
> Following an older tutorial? Before OpenTelemetry Python 1.40.0 you had to set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to turn on log instrumentation. In current versions you don't, as long as opentelemetry-instrumentation-logging is installed.
Step 4: Run it with auto-instrumentation
Set a few environment variables, then start the app through opentelemetry-instrument.
macOS and Linux:
export OTEL_SERVICE_NAME=orders-api
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_METRIC_EXPORT_INTERVAL=5000
opentelemetry-instrument uvicorn main:app --port 8000
Windows (PowerShell):
$env:OTEL_SERVICE_NAME = "orders-api"
$env:OTEL_TRACES_EXPORTER = "otlp"
$env:OTEL_METRICS_EXPORTER = "otlp"
$env:OTEL_LOGS_EXPORTER = "otlp"
$env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317"
$env:OTEL_METRIC_EXPORT_INTERVAL = "5000"
opentelemetry-instrument uvicorn main:app --port 8000
What the variables do:
| Variable | What it does |
| --- | --- |
| OTEL_SERVICE_NAME | The name shown in Grafana. Skip it and your data shows up as unknown_service. |
| OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER | Where each signal goes. otlp means send it over OTLP. |
| OTEL_EXPORTER_OTLP_ENDPOINT | Where the backend listens. This is the gRPC default, shown so you know where to change it. |
| OTEL_METRIC_EXPORT_INTERVAL | Milliseconds between metric exports. Set low here so metrics show up quickly (more on that below). |
Your code contains no SDK setup, no exporter wiring, no FastAPI hooks and no log handler.
opentelemetry-instrument did all of that before your app started. It also creates the POST /orders span and some HTTP metrics for every request.
> Skip --reload and --workers for now. The auto-reloader spawns child processes and can break instrumentation. And with uvicorn --workers, the official troubleshooting guide says you get traces and metrics but no logs. For several workers, it recommends Gunicorn with Uvicorn workers, which exports all three.
Step 5: Send some traffic
In a second terminal, create load.py:
import json
import random
import urllib.error
import urllib.request
URL = "http://localhost:8000/orders"
for _ in range(40):
body = json.dumps({
"item": random.choice(["keyboard", "mouse", "monitor"]),
"quantity": random.randint(1, 3),
}).encode()
request = urllib.request.Request(URL, data=body, headers={"Content-Type": "application/json"})
try:
urllib.request.urlopen(request)
print("paid")
except urllib.error.HTTPError as error:
print("declined" if error.code == 402 else f"error {error.code}")
Run it a couple of times:
python load.py
A stream of paid and declined scrolls past. Roughly one in four orders is declined.
Step 6: Explore it in Grafana
Open Grafana and use Explore from the left menu. Pick a data source at the top, one signal at a time. Menu names move around between Grafana versions, so don't worry if yours looks slightly different.
Traces (Tempo)
Choose the Tempo data source and search for service orders-api. Open any trace.
A waterfall appears: POST /orders at the top, process_order under it, charge_card under that. Each bar's length is how long that step took.
Now find a declined one. Its charge_card span is marked as an error, with the exception attached.
Metrics (Prometheus)
Choose Prometheus and try these queries:
sum by (outcome) (rate(orders_processed_total[1m]))
histogram_quantile(0.95, sum by (le) (rate(payment_duration_milliseconds_bucket[1m])))
The first shows paid versus declined orders per second. The second shows the 95th percentile payment time.
Prometheus's default OTLP translation adjusts metric names: dots become underscores, counters gain _total, and the unit becomes a suffix (ms turns into milliseconds). If autocomplete shows a slightly different name, your setup uses another translation setting, so use what it shows.
Logs (Loki)
Choose Loki and run:
{service_name="orders-api"}
Your "Processing order" and "Payment declined" lines show up. Click one to expand it, and look for its trace ID.
Follow one failure across all three
This is the payoff. Try it:
1. In Prometheus, spot the declined rate.
2. In Tempo, open a trace with a red charge_card span.
3. Jump from that span to its logs, and read exactly what happened.
The image's Grafana comes pre-wired so you can click from a trace to its logs, and from a log line back to its trace.
If a link is missing in your version, query Loki directly. With Loki's native OTLP ingestion, the service name is an index label and the trace ID is structured metadata. So you can filter on the ID you copied from Tempo:
{service_name="orders-api"} | trace_id="PASTE_THE_TRACE_ID_HERE"
That's one request, seen through all three signals.
Why your metrics showed up later than your traces
Each signal is batched and sent on its own timer:
| Signal | Default delay between exports | Variable |
| --- | --- | --- |
| Logs | 1 second | OTEL_BLRP_SCHEDULE_DELAY |
| Traces | 5 seconds | OTEL_BSP_SCHEDULE_DELAY |
| Metrics | 60 seconds | OTEL_METRIC_EXPORT_INTERVAL |
Without the 5-second setting we used, you'd wait up to a minute to see your first metric. It's a classic first-day surprise.
Do you need your own Collector?
In this setup, the Collector inside the Grafana image does the job. You'll want your own as you grow.
The OpenTelemetry docs give a few reasons. It gives many services one shared place to send data. It can gather traces across services running on different hosts. And it's a central spot to process data before it reaches a backend.
It also keeps backend details out of your apps, so you can switch backends without touching your code.
Their advice is that unless you have a single service or you're just experimenting, you'll want a Collector in production.
A handy debugging trick
When data isn't arriving, run a bare Collector that just prints what it receives. Save this as otel-collector-config.yaml:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [debug]
metrics:
receivers: [otlp]
exporters: [debug]
logs:
receivers: [otlp]
exporters: [debug]
Stop the Grafana container first, since both want the same ports. Then run:
docker run -p 4317:4317 -p 4318:4318 \
-v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
otel/opentelemetry-collector:latest \
--config=/etc/otel-collector-config.yaml
On Windows, use the full path to the file instead of $(pwd).
Restart your app and send traffic. Every span, metric and log shows up in the Collector's output. If you see them there, your app is fine and the problem is further downstream.
Common first-day problems
Most of these are explained above, so this is just an index:
| Symptom | Where to look |
| --- | --- |
| No data anywhere, and no errors | The app wasn't started through opentelemetry-instrument, or the endpoint is wrong. Run the debug Collector above to check. |
| Metrics are missing at first | The export delay table above |
| INFO logs are missing | The logging gotcha in Step 3 |
| Logs vanish with --workers, or everything vanishes with --reload | The callout in Step 4 |
| Everything disappeared | The note in Step 1 about where the image keeps its data |
From demo to production
The demo is deliberately simple. A real setup adds a few things:
- A real backend. Pick a managed service or a properly run self-hosted stack in place of the demo image.
- Sampling. You usually don't keep every trace. Setting OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 keeps about 10%.
- Care with sensitive data. Don't put passwords, tokens or personal data into attributes or log messages. The OpenTelemetry docs have a page on handling sensitive data.
- Consistent names. OpenTelemetry has semantic conventions, a shared vocabulary for attribute names like http.route. Following them keeps your data readable across tools.
And when one service calls another over HTTP, the instrumentation passes the trace context along in headers. Both services then show up in one trace. That's context propagation, and it's the feature that makes tracing really shine across a system.
Where to go next
- The OpenTelemetry Demo. A multi-service shop with telemetry already wired in. It's the best place to see tracing across services.
- Manual instrumentation for Python. More on spans, events, links and metric instruments.
- Collector processors. Filter, transform and sample data before it leaves.
- More libraries. Add httpx, SQLAlchemy or a database driver, then rerun opentelemetry-bootstrap -a install. It picks up new libraries on its own.
Recap
- OpenTelemetry produces telemetry. A backend stores and shows it.
- Traces show one request's journey, metrics show trends, logs tell the story. Their power comes from being linked.
- opentelemetry-instrument gives you request spans and log shipping for free. You add spans and metrics for your own business logic.
- Set OTEL_SERVICE_NAME, and shorten the metric interval while you're learning.
- Keep metric attributes low-cardinality. Put the detail on spans.
- The otel-lgtm image is a great sandbox, not a production backend.
Instrument, export, then follow one request across all three signals. Everything else builds on that.
- * *
_Details checked against the OpenTelemetry documentation and the grafana/docker-otel-lgtm project on October 25, 2026. Tooling moves quickly, so if a command or menu behaves differently on your version, the official docs win._
Sources
- OpenTelemetry Python: Getting Started by Example
- OpenTelemetry Python: status and releases
- Python logs auto-instrumentation example
- Troubleshooting Python automatic instrumentation
- OpenTelemetry SDK environment variables
- grafana/docker-otel-lgtm on GitHub
- Grafana: An OpenTelemetry backend in a Docker image
---
Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
1 reaction
【Vol.1】Python特化のOpenTelemetry＋LGTMスタック最小構成
1. はじめに
OpenTelemetryの学習にあたり、公式デモを、ようやく動かせるようになったので、それを忘れないうちに、Python（Flask/Locust）特化のシンプルな最小構成環境を作ってみました。
手作業を排除し、EC2上にインフラ構築から自動監視までをTerraformで自動作成します。
:::note info
今回は 【Vol.1】環境構築編 として、まずは動かすことに特化しています。
自分自身の学習記録の備忘録としての意味合いもあります。
:::
!image.png
2. Grafana Labs公式「LGTMスタック」デモとは、
本環境は、Grafana Labs公式の OpenTelemetry Demo をベースにしています。公式デモは、モダンな可観測性（Observability）の標準である 「LGTMスタック」 を手軽に体験できる素晴らしい環境です。
📊 LGTMスタックの構成要素
| コンポーネント | 役割 |
| :--- | :--- |
| Loki | ログを集約・検索する軽量なログ管理システム |
| Grafana | 収集データを1つにまとめて可視化するダッシュボード |
| Tempo | リクエストの経路を追跡する分散トレーシング |
| Mimir / Prometheus | システム状態（CPUやリクエスト数など）を蓄積するメトリクス |
:::note indfo
「LGTMスタック」イメージのみ流用し、アプリ側を 「Python（Flask/Locust）の最小構成」 に絞り込み、ファイル構成、設定内容を理解しやすくしました。
:::
3. 今回のチャレンジと「あえてやらなかったこと」
今回の最大の目的は、 「必要な部分だけを切り出して、自分なりにカスタマイズしやすい環境を作る」 ことです。
具体的には、以下の要素に絞って手を動かし、理解を深めました。
- Grafanaでの可視化の仕組み
- ダッシュボードが「何を、どう表示しているか」の整理
- PromQLや関数を用いたグラフ描画の分析
- OpenTelemetry（メトリクス中心）の挙動確認
- アプリからデータが「どのように出力されるか」の観察
- 負荷（Locust）をかけたときのメトリクス変動の観測
:::note info
「あえてやらなかったこと」は、LGTMスタック各コンポーネント（Loki, Tempo, Mimir, OTel Collector）間の内部接続や詳細設定は公式デモの構成を流用しています。
:::
4. 技術スタック
本環境で使用している技術スタックは以下の通りです。
- インフラ（IaC）：Terraform
- クラウド：AWS (VPC, Subnet, Security Group, EC2 [Amazon Linux 2023])
- コンテナ：Docker / Docker Compose
- アプリケーション：Python 3.11 / Flask (Webアプリ) / Locust (負荷テストツール)
- 可観測性（Observability）エコシステム：
- データ収集：OpenTelemetry
- 可視化：Grafana
- メトリクス蓄積：Prometheus
- ログ蓄積：Loki
- 分散トレーシング：Tempo
ディレクトリ構成
本環境の全体構成は以下の通りです。
このデータセットは、GitHubリポジトリ に登録しています。
├── ENV_1011/ # main.tfの構築・プロビジョニング時に使用される環境データ
│ ├── my-dashboards/ # Grafanaのダッシュボード設定を格納
│ │ └── grafana-dashboard-python-metrics.json
│ ├── python/ # OpenTelemetryによるメトリクス生成を組み込んだアプリ
│ │ ├── app.py # Flaskアプリケーション本体
│ │ └── locustfile.py # 負荷試験用のLocustスクリプト
│ ├── services/ # EC2上で常時稼働させるsystemd用の設定ファイル
│ │ ├── flask-app.service
│ │ └── locust-worker.service
│ ├── clear-defaults.yaml # 各種環境の初期設定をクリア・調整するための設定
│ ├── my-dashboard-config.yaml # Grafanaダッシュボードのプロバイダー設定
│ └── run-lgtm.sh # LGTMスタック（Grafana等）を起動・制御するスクリプト
└── devops_1011/ # 構築・自動化用コード
├── scripts/
│ └── setup.sh # お試し構築・初期セットアップ用スクリプト
└── terraform/ # IaC（Infrastructure as Code）関連コード
├── main.tf # AWS EC2インフラやネットワークの定義
└── output.tf # 構築後のパブリックIP等の出力定義
各ディレクトリ・ファイルの役割
>### インフラ自動化・構築 (./devops_1011/)
Terraformを用いて、AWSのEC2上にインフラ環境を自動構築します。
!image.png
>### アプリケーション & 自動監視環境 ( ./ENV_1011/ )
main.tf を実行し、EC2インスタンス起動後に、環境構築/python実行で必要なファイルを格納しています。
!image.png
5. システム構成
!image.png
今回はGrafana Labs公式デモ版のコンテナイメージ（grafana/otel-lgtm）をベースに使用しているため、OpenTelemetry Collector部や各バックエンド（Prometheus / Tempo / Loki / Pyroscope）の内部構成はそちらに依存しています。
このシステムに対して、自作した「Python Web App (Flask)」と「負荷生成器（Locust）」を接続し、EC2インスタンス（t3.small）内でsystemdとしてバックグラウンド実行させています。
インフラ構築や環境配備の自動化を図るために、Terraform でAWSリソースの自動生成を行い、EC2のUser DataをトリガーにGitHub・Docker Hubと連携してデプロイを自動化しています。
- GitHub：最新のソースコード（app.py / locustfile.py）を自動取得
- Docker Hub：可視化基盤のコンテナイメージ（grafana/otel-lgtm）をプル
🔄 各コンポーネントの役割
!image.png
6. メトリクス
ダッシュボードで可視化したメトリクスは以下の通りです。
Locustの負荷変動に応じて、これらの値も変動します。
!image.png
具体的なコードについては、末尾の [【付録：環境構築用コード】](#10付録環境構築用コード) にある「grafana-dashboard-python-metrics.json」を参照してください。
:::note info
今回の構成ではメトリクス収集に特化しており、LokiやTempoへのデータ送信に必要な機能実装は行わなかったので、 ログ、トレース の内容は見られません。
:::
7.デプロイ手順
インフラのプロビジョニングからアプリのデプロイ、監視の有効化まで、Terraformで自動化しています。
>### 1. 環境変数の設定
ローカル環境に自身のAWSの認証情報（アクセスキーなど）をセットします。
>### 2. Terraformの実行
terraform apply を実行します。AWS上にVPC、セキュリティグループ、EC2（Amazon Linux 2023）が自動生成されます。
※具体的なIaCコードは末尾の [【付録：環境構築用コード】](#10付録環境構築用コード) を参照。
>### 3. user_dataによる完全自動セットアップ
EC2の起動時、user_dataがバックグラウンドで以下の処理を全自動で実行します。
1. GitのSparse Checkout（部分チェックアウト） を使い、リポジトリから必要なディレクトリ（ENV_1011）のみダウンロード。
2. Docker / Docker Compose のインストールと起動。
3. 公式スクリプト（run-lgtm.sh）によるLGTMスタックコンテナ起動。
4. Python 3.11環境の構築と、必要ライブラリ（Flask, OpenTelemetry SDK, Locustなど）のインストール。
5. Pythonスクリプトの配置、およびsystemdへのサービス登録と自動起動。
>### 4. Grafanaへのアクセス
terraform apply 完了後、数分待つだけでセットアップが完了します。出力されたパブリックIPの3000番ポート（http://<EC2のパブリックIP>:3000）へアクセスすれば、すぐに完成したダッシュボードを確認できます。
<details>
<summary>詳細手順は、こちらを参照</summary>
>### URL指定(Grafana-GUI)
terraform apply 完了後、以下の様にパブリックIPが出力されます。
ec2_private_ip = "10.0.1.206"
ec2_public_ip = "3.112.205.128"
上記のケースでは、以下のURL に2～3分待ってアクセスします。
3.112.205.128
>### ダッシュボード選択
以下の様に、Grafana からダッシュボード選択を行います。
!image.png
>### メニュー選択
表示期間を「⏱ Last 5 minutes」に切り替えて、10分ほど待つと、以下のようなデータが見られるようになります。
!image.png
</details>
8. おわりに
今回の取り組みを通じ、OpenTelemetryの強力さとLGTMスタックによる監視基盤の構築方法を深く学ぶことができました。
手間の多い手動構築（SSH、クローン、権限変更、サービス登録など）を、Terraformの user_data と Gitの Sparse Checkout で自動化し、「コマンドで環境構築」する面白味を実感できました。
>### 周辺技術での学び
環境コード化にあたり、監視以外の実用的なスキルも習得できました。
- Locustによる動的負荷試験：リクエストを変動させ、意図的にメトリクスの波（トレンド）を作る手法
- Terraformの user_data：EC2起動時のプロビジョニングを完全自動化する実務的な手法
9. 今後の展望
今回は【Vol.1】として「最小構成で動かす」ことを最優先しました。このベース環境を活かし、以下の要素にも挑戦する予定です。(あくまでも予定です)
- 監視基盤の自作化：公式デモの流用を脱却し、各コンポーネントとCollector間の接続・設定をイチから自作する
- 自動制御の組み込み：各バックエンドのデータをシェルスクリプト等で定期取得し、異常を自動検知する仕組みの実装。
- 構成管理の高度化：user_data（シェルスクリプト）で行っているアプリ配置やサービス登録を、Ansibleによる管理へ移行する。
- オブザーバビリティの完全化：今回はメトリクスのみのだった構成にLoki（ログ）とTempo（トレース）の実装を本格追加し、「3つの柱」を可視化できる環境の構築。
10.付録：環境構築用コード
GitHubレポジトリ に登録しています
<details>
<summary>./terraform/main.tf</summary>
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = "ap-northeast-1"
}
# 1. VPC（仮想ネットワーク）の作成
resource "aws_vpc" "main" {
cidr_block = "10.0.0.0/16"
enable_dns_hostnames = true
tags = {
Name = "study-vpc"
}
}
# 2. パブリックサブネットの作成
resource "aws_subnet" "public" {
vpc_id = aws_vpc.main.id
cidr_block = "10.0.1.0/24"
map_public_ip_on_launch = true # 起動時にパブリックIPを自動割り当て
tags = {
Name = "study-public-subnet"
}
}
# 3. インターネットゲートウェイの作成（外の世界と繋ぐ窓口）
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
tags = {
Name = "study-igw"
}
}
# 4. ルートテーブルの作成と紐付け（ネットへの道を登録）
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
tags = {
Name = "study-public-rt"
}
}
resource "aws_route_table_association" "public" {
subnet_id = aws_subnet.public.id
route_table_id = aws_route_table.public.id
}
# =================================================================================
# 1. セキュリティグループの定義
# =================================================================================
resource "aws_security_group" "ec2" {
name = "study-ec2-sg"
description = "Security Group for OpenTelemetry and App Server"
vpc_id = aws_vpc.main.id # ※環境に合わせてVPC IDの変数名を調整してください
# -------------------------------------------------------------------------------
# 各種アプリケーション・可観測性ツール用のインバウンドルール（dynamicブロックで一括管理）
# -------------------------------------------------------------------------------
dynamic "ingress" {
for_each = {
22 = "SSH"
3000 = "Grafana UI"
16686 = "Jaeger UI"
9090 = "Prometheus UI"
3100 = "Loki"
8000 = "Python Web App"
9464 = "OpenTelemetry Prometheus Exporter"
5000 = "Python App (Flask)"
4317 = "OpenTelemetry (OTLP/gRPC)"
4318 = "OpenTelemetry (OTLP/HTTP)"
8080 = "OpenTelemetry Demo Web & Grafana"
}
content {
description = ingress.value
from_port = ingress.key
to_port = ingress.key
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
# -------------------------------------------------------------------------------
# 【超重要】EC2同士（VPC内部）の全通信を許可するルール
# -------------------------------------------------------------------------------
ingress {
description = "VPC Internal All Traffic"
from_port = 0
to_port = 0
protocol = "-1" # すべてのプロトコルを許可
cidr_blocks = ["10.0.0.0/16"] # VPC内部のIPアドレスからの通信をすべて許可
}
# -------------------------------------------------------------------------------
# インターネットへの外向き通信をすべて許可（重複を排除して1つにスッキリ整理）
# -------------------------------------------------------------------------------
egress {
description = "Allow all outbound traffic"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
# =================================================================================
# 2. SSH接続用のキーペアを自動生成
# =================================================================================
resource "tls_private_key" "keygen" {
algorithm = "RSA"
rsa_bits = 4096
}
resource "aws_key_pair" "keypair" {
key_name = "study-key"
public_key = tls_private_key.keygen.public_key_openssh
}
# 生成された秘密鍵（Terra Termで使う.pemファイル）をローカルに保存
resource "local_file" "private_key" {
content = tls_private_key.keygen.private_key_pem
filename = "${path.module}/study-key.pem"
}
# =================================================================================
# 3. AWSから自動で最新のAmazon Linux 2023 AMI IDを取得
# =================================================================================
data "aws_ssm_parameter" "amzn_linux_2023" {
name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}
# =================================================================================
# 4. EC2インスタンスの作成（GitHubパブリック方式・完全自動化版）
# =================================================================================
resource "aws_instance" "app_server" {
ami = data.aws_ssm_parameter.amzn_linux_2023.value
instance_type = "t3.small"
subnet_id = aws_subnet.public.id
vpc_security_group_ids = [aws_security_group.ec2.id]
key_name = aws_key_pair.keypair.key_name
# ルートディスク容量を30GB（無料枠の最大値）に増やす設定
root_block_device {
volume_size = 30
volume_type = "gp3"
}
tags = {
Name = "study-ec2-1"
}
# ===============================================================================
# 【完全自動化】初期化シェルスクリプト（user_data）
# ===============================================================================
user_data = <<-EOF
#!/bin/bash
# エラーが発生したら即座にスクリプトを終了させる設定
set -euo pipefail
echo "=========================================================="
echo "=== 1. GitHubから特定のディレクトリ（ENV_1011）のみを取得 ==="
echo "=========================================================="
dnf install -y git
sleep 5
mkdir -p /opt/deploy_data
cd /opt/deploy_data
# Sparse Checkout（部分チェックアウト）の設定
git init
git remote add origin https://github.com/wata123-t/deploy_data
git config core.sparseCheckout true
echo "ENV_1011/" >> .git/info/sparse-checkout
# データの引っ張りと移動
git pull origin main
cd ENV_1011
echo "=========================================================="
echo "=== 2. Docker & Docker Compose のインストールと起動 ==="
echo "=========================================================="
dnf install -y docker
sleep 5
systemctl enable --now docker
# Docker Compose V2 のインストール
mkdir -p /usr/local/lib/docker/cli-plugins/
curl -SL https://github.com/docker/compose/releases/download/v2.29.2/docker-compose-linux-x86_64 -o /usr/local/lib/docker/cli-plugins/docker-compose
chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
echo "=========================================================="
echo "=== 3. Grafana Labs公式「LGTMスタック」デモの実行 ==="
echo "=========================================================="
# すでにGit側で権限を付与していますが、念のため再付与して実行
chmod +x ./run-lgtm.sh
./run-lgtm.sh
sleep 10
echo "=========================================================="
echo "=== 4. Python 3.11 & 開発ツールのインストール ==="
echo "=========================================================="
dnf install -y python3.11 python3.11-pip python3.11-devel gcc
sleep 10
echo "=========================================================="
echo "=== 5. 必要なPythonライブラリの一括インストール ==="
echo "=========================================================="
python3.11 -m pip install --upgrade pip
sleep 10
python3.11 -m pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-system-metrics locust
sleep 10
echo "=========================================================="
echo "=== 6. アプリケーションフォルダの作成とファイルのコピー ==="
echo "=========================================================="
mkdir -p /opt/script/python/app
# GitHubから落としてきた相対パスからファイルをコピー
cp "./python/app.py" /opt/script/python/app/app.py
cp "./python/locustfile.py" /opt/script/python/locustfile.py
echo "=========================================================="
echo "=== 7. 各種サービス（systemd）ファイルの配置と起動 ==="
echo "=========================================================="
cp "./services/flask-app.service" /etc/systemd/system/flask-app.service
cp "./services/locust-worker.service" /etc/systemd/system/locust-worker.service
systemctl daemon-reload
systemctl enable --now flask-app locust-worker
echo "=========================================================="
echo "=== すべてのセットアップ処理が正常に完了しました！ ==="
echo "=========================================================="
EOF
}
</details>
<details>
<summary>./terraform/output.tf</summary>
# =================================================================================
# 5. アウトプット（接続用IPアドレスの表示）
# =================================================================================
output "ec2_public_ip" {
value = aws_instance.app_server.public_ip
description = "起動したEC2のパブリックIPアドレスです"
}
output "ec2_private_ip" {
value = aws_instance.app_server.private_ip
description = "EC2のプライベートIPアドレスです"
}
</details>
<details>
<summary>./my-dashboards/grafana-dashboard-python-metrics.json
</summary>
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"description": "Dashboard for **Python** metrics with OpenTelemetry instrumentation",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 18812,
"graphTooltip": 0,
"links": [
{
"asDropdown": false,
"icon": "info",
"includeVars": false,
"keepTime": false,
"tags": [],
"targetBlank": false,
"title": "Semantic Conventions: 1.20.0 - 1.22.0",
"tooltip": "multiple versions of the semantic conventions are supported using 'or' and regex queries",
"type": "link",
"url": "https://github.com/open-telemetry/semantic-conventions/blob/main/schemas/1.20.0"
}
],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "HTTP server request rate",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 0
},
"id": 51,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate({__name__=~\"http_server_request_duration_seconds_count|http_server_request_duration_count|http_server_duration_seconds_count|http_server_duration_count|http_server_duration_milliseconds_count|http_server_requests_milliseconds_count|http_server_requests_count|http_server_requests_seconds_count\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))",
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "Rate",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "HTTP server error ratio - ratio of requests that return 5xx",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 0
},
"id": 50,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "(sum(rate(http_server_duration_milliseconds_count{http_status_code=~\"5..\"}[$__rate_interval])) / sum(rate(http_server_duration_milliseconds_count[$__rate_interval]))*1) or on() vector(0)",
"instant": false,
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "Error %",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "95th percentile of HTTP server request duration in seconds",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "s"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 0
},
"id": 52,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "(histogram_quantile(0.95, sum by(le, instance) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket|http_server_requests_seconds_bucket\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))))\nor\n(histogram_quantile(0.95, sum by(le, instance) (rate({__name__=~\"http_server_duration_milliseconds_bucket|http_server_requests_milliseconds_bucket|http_server_requests_bucket\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))) / 1000)\n",
"instant": false,
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Duration (95%)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "CPU utilization for the whole system",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 7,
"w": 13,
"x": 0,
"y": 8
},
"id": 38,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "process_cpu_utilization_ratio",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "CPU utilization",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "Used heap memory / heap memory limit ",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 7,
"w": 11,
"x": 13,
"y": 8
},
"id": 30,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "process_memory_usage_bytes",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Heap Memory utilization",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "Percentage of time spend for garbage collection pauses",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "bar",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 7,
"w": 13,
"x": 0,
"y": 15
},
"id": 46,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum by (instance) (rate(cpython_gc_collections_total[$__rate_interval]))*60",
"hide": false,
"legendFormat": "__auto",
"range": true,
"refId": "B"
}
],
"title": "Garbage Collection",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"description": "Number of currently executing (also called \"live\") threads",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 25,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 7,
"w": 11,
"x": 13,
"y": 15
},
"id": 42,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "process_thread_count",
"legendFormat": "{{instance}}",
"range": true,
"refId": "A"
}
],
"title": "Threads",
"type": "timeseries"
}
],
"refresh": "",
"schemaVersion": 38,
"tags": ["Python"],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "prometheus"
},
"description": "Choose a Prometheus data source",
"hide": 0,
"includeAll": false,
"label": "Data source",
"multi": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"queryValue": "",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
},
{
"allValue": ".+",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(http_server_request_duration_seconds_count, job)",
"hide": 0,
"includeAll": true,
"label": "Job",
"multi": true,
"name": "job",
"options": [],
"query": {
"query": "label_values(http_server_request_duration_seconds_count, job)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": ".+",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(http_server_request_duration_seconds_count, instance)",
"description": "The instance of the application, e.g. pod1",
"hide": 0,
"includeAll": true,
"label": "Instance",
"multi": true,
"name": "instance",
"options": [],
"query": {
"query": "label_values(http_server_request_duration_seconds_count, instance)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 2,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"tagValuesQuery": "",
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d"]
},
"timezone": "",
"title": "python-Objability",
"uid": "python-dashboard-metrics",
"version": 1,
"weekStart": ""
}
</details>
1 like
Mentions
OpenTelemetry for Beginners: Logs, Metrics and Traces in One Setup Picture this. A customer says checkout feels slow. You open the logs and find thousands of lines. Which request was theirs? Which step was slow? Was it just them, or everyone? Logs alone can't answer all of that. You need three kinds of data about the same request, and a way to connect them. That's what OpenTelemetry gives you. It has a reputation for being complicated, but the core idea is small. In this post you'll send traces, metrics and logs from a small FastAPI app into Grafana. The backend is one Docker command. The boilerplate is handled by zero-code instrumentation. You write a few lines yourself for the interesting parts. > Heads up on Python logs: In OpenTelemetry Python, traces and metrics are marked Stable, but logs are still marked Development. Logging works fine for this walkthrough. Just expect the logs side to change more than the other two. What OpenTelemetry is (and what it isn't) OpenTelemetry, or OTel for short, is an open standard and a set of tools for producing telemetry. That's the traces, metrics and logs your software emits about itself. It's a Cloud Native Computing Foundation project, and it grew out of two older projects, OpenTracing and OpenCensus. The key word is _producing_. OpenTelemetry doesn't store or display anything. You still need a backend for that. The win is that you instrument once, and your code doesn't care which backend receives the data. Here are the pieces you'll meet: | Piece | What it does | In this post | | --- | --- | --- | | API and SDK | The code you call, and the engine behind it that batches and exports data | Installed with opentelemetry-distro | | Instrumentation | Libraries that create telemetry for you (web frameworks, HTTP clients, databases) | FastAPI and logging | | OTLP | The format and protocol the data travels in | gRPC on port 4317 | | Collector | A service that receives, processes and forwards telemetry | Built into the Grafana image | | Backend | Where the data is stored and viewed | Tempo, Prometheus, Loki and Grafana | The three signals in plain words | Signal | Plain-English meaning | Question it answers | Example | | --- | --- | --- | --- | | Traces | The story of one request as it moves through your code | Where did this request spend its time? | POST /orders took 380 ms, and 340 of that was charge_card | | Metrics | Numbers measured over time | How often? How slow? Is it getting worse? | Declined orders per minute | | Logs | Timestamped messages, in words | What exactly happened? | Payment declined: keyboard | Two terms you'll see everywhere. A span is one step inside a trace, like "charge the card". A trace is a tree of spans. Spans carry attributes, which are key-value details such as order.item = keyboard. Which signal do you reach for first? It depends on the question: !Mermaid Diagram The real power is in the links. A metric spike leads you to a trace. A span leads you to the logs written during it. That last jump works because each log record carries the trace ID and span ID of the request that wrote it. OpenTelemetry adds them for you. What we'll build A tiny orders API with one endpoint, POST /orders. It "charges a card", which is just a short sleep with a 25% chance of being declined. That gives you slow requests and failures to look at. Here's how the data flows: !Mermaid Diagram Each request will produce a trace shaped like this: !Mermaid Diagram The official docs make a useful point here. Automatic instrumentation captures what happens at the edges of your system, like inbound and outbound HTTP requests. It can't see what's going on inside your own code. So the top span is free. The two inner spans, the counter and the histogram are yours. You'll need Python 3.10 or newer (that's what OpenTelemetry Python supports), Docker, and a terminal. Step 1: Start the backend docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm This is Grafana's otel-lgtm image. It bundles an OpenTelemetry Collector, Prometheus for metrics, Tempo for traces, Loki for logs and Grafana to look at all three. The ports: - 3000 is the Grafana UI - 4317 is where OTLP over gRPC arrives - 4318 is where OTLP over HTTP arrives Open localhost and log in with user admin and password admin. > Development only: Grafana says this image is intended for development, demo and testing. The data lives inside the container, so it's gone when the container is. Don't use it as your production backend. Step 2: Set up the project mkdir orders-api && cd orders-api python3 -m venv venv source venv/bin/activate pip install fastapi "uvicorn[standard]" opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-logging opentelemetry-bootstrap -a install On Windows PowerShell, activate with .\venv\Scripts\Activate.ps1 and use python instead of python3. What each piece does: - opentelemetry-distro brings the API and SDK, plus two commands: opentelemetry-bootstrap and opentelemetry-instrument. - opentelemetry-exporter-otlp lets the app send data over OTLP. - opentelemetry-instrumentation-logging connects Python's logging module to OpenTelemetry. - opentelemetry-bootstrap -a install looks at what's installed and adds the matching instrumentation. Here, that means FastAPI. Run bootstrap _after_ you install FastAPI, because it decides based on what it finds. Step 3: Write the app Create main.py: import asyncio import logging import random import time from fastapi import FastAPI, HTTPException from opentelemetry import metrics, trace from pydantic import BaseModel, Field app = FastAPI(title="orders-api") # Let INFO records through. logger = logging.getLogger("orders") logger.setLevel(logging.INFO) tracer = trace.get_tracer("orders.tracer") meter = metrics.get_meter("orders.meter") orders_processed = meter.create_counter( "orders.processed", description="Orders processed, by outcome", ) payment_duration = meter.create_histogram( "payment.duration", unit="ms", description="Time spent charging the card", ) class Order(BaseModel): item: str quantity: int = Field(ge=1, le=10) class PaymentDeclined(Exception): """Raised when the pretend payment provider says no.""" async def charge_card(amount: float) -> None: started = time.perf_counter() with tracer.start_as_current_span("charge_card") as span: span.set_attribute("payment.amount", amount) try: await asyncio.sleep(random.uniform(0.05, 0.4)) # pretend to call a payment API if random.random() < 0.25: raise PaymentDeclined("card declined") finally: payment_duration.record((time.perf_counter() - started) * 1000) @app.post("/orders") async def create_order(order: Order): with tracer.start_as_current_span("process_order") as span: span.set_attribute("order.item", order.item) span.set_attribute("order.quantity", order.quantity) logger.info("Processing order: %s x%d", order.item, order.quantity) try: await charge_card(round(order.quantity * 9.99, 2)) outcome = "paid" logger.info("Order paid: %s", order.item) except PaymentDeclined: outcome = "declined" logger.warning("Payment declined: %s", order.item) span.set_attribute("order.outcome", outcome) orders_processed.add(1, {"outcome": outcome}) if outcome == "declined": raise HTTPException(status_code=402, detail="Payment declined") return {"status": "paid", "item": order.item} You don't need to memorise any of it. These are the parts worth understanding. Spans tracer.start_as_current_span(...) opens a span for the length of the with block. A span opened inside another one becomes its child. That's how charge_card ends up under process_order. set_attribute attaches details. Put anything here that will help you debug later, like the item or the amount. There's a nice default too. If an exception escapes a with span block, the SDK records it on the span and marks the span as an error. That's why a declined payment will show up as a red charge_card span, with no extra code from you. Metrics Two instruments, two jobs: - Counter: counts things, and only goes up. We add 1 for every order, labelled with its outcome. - Histogram: records measurements like durations. It lets you ask for percentiles later, such as "how slow are the slowest 5%?" I record the duration in milliseconds on purpose. The default histogram buckets are tuned for millisecond-sized values: 5, 10, 25, 50 and so on up to 10,000. Record seconds instead and almost everything lands in the first bucket, so your percentiles turn to mush. > Careful with metric attributes: Every unique combination of attribute values creates a new time series. outcome has two values, which is fine. A user ID or order ID would create millions. Keep high-detail values like order.item on spans, where they're cheap. Logs There's no special OpenTelemetry logging call here. You keep using Python's logging, and OpenTelemetry attaches a handler that ships the records. One gotcha: Python only lets WARNING and above through by default. So we set our logger to INFO, or the "Processing order" lines would silently vanish. > Following an older tutorial? Before OpenTelemetry Python 1.40.0 you had to set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true to turn on log instrumentation. In current versions you don't, as long as opentelemetry-instrumentation-logging is installed. Step 4: Run it with auto-instrumentation Set a few environment variables, then start the app through opentelemetry-instrument. macOS and Linux: export OTEL_SERVICE_NAME=orders-api export OTEL_TRACES_EXPORTER=otlp export OTEL_METRICS_EXPORTER=otlp export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 export OTEL_METRIC_EXPORT_INTERVAL=5000 opentelemetry-instrument uvicorn main:app --port 8000 Windows (PowerShell): $env:OTEL_SERVICE_NAME = "orders-api" $env:OTEL_TRACES_EXPORTER = "otlp" $env:OTEL_METRICS_EXPORTER = "otlp" $env:OTEL_LOGS_EXPORTER = "otlp" $env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317" $env:OTEL_METRIC_EXPORT_INTERVAL = "5000" opentelemetry-instrument uvicorn main:app --port 8000 What the variables do: | Variable | What it does | | --- | --- | | OTEL_SERVICE_NAME | The name shown in Grafana. Skip it and your data shows up as unknown_service. | | OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER | Where each signal goes. otlp means send it over OTLP. | | OTEL_EXPORTER_OTLP_ENDPOINT | Where the backend listens. This is the gRPC default, shown so you know where to change it. | | OTEL_METRIC_EXPORT_INTERVAL | Milliseconds between metric exports. Set low here so metrics show up quickly (more on that below). | Your code contains no SDK setup, no exporter wiring, no FastAPI hooks and no log handler. opentelemetry-instrument did all of that before your app started. It also creates the POST /orders span and some HTTP metrics for every request. > Skip --reload and --workers for now. The auto-reloader spawns child processes and can break instrumentation. And with uvicorn --workers, the official troubleshooting guide says you get traces and metrics but no logs. For several workers, it recommends Gunicorn with Uvicorn workers, which exports all three. Step 5: Send some traffic In a second terminal, create load.py: import json import random import urllib.error import urllib.request URL = "http://localhost:8000/orders" for _ in range(40): body = json.dumps({ "item": random.choice(["keyboard", "mouse", "monitor"]), "quantity": random.randint(1, 3), }).encode() request = urllib.request.Request(URL, data=body, headers={"Content-Type": "application/json"}) try: urllib.request.urlopen(request) print("paid") except urllib.error.HTTPError as error: print("declined" if error.code == 402 else f"error {error.code}") Run it a couple of times: python load.py A stream of paid and declined scrolls past. Roughly one in four orders is declined. Step 6: Explore it in Grafana Open Grafana and use Explore from the left menu. Pick a data source at the top, one signal at a time. Menu names move around between Grafana versions, so don't worry if yours looks slightly different. Traces (Tempo) Choose the Tempo data source and search for service orders-api. Open any trace. A waterfall appears: POST /orders at the top, process_order under it, charge_card under that. Each bar's length is how long that step took. Now find a declined one. Its charge_card span is marked as an error, with the exception attached. Metrics (Prometheus) Choose Prometheus and try these queries: sum by (outcome) (rate(orders_processed_total[1m])) histogram_quantile(0.95, sum by (le) (rate(payment_duration_milliseconds_bucket[1m]))) The first shows paid versus declined orders per second. The second shows the 95th percentile payment time. Prometheus's default OTLP translation adjusts metric names: dots become underscores, counters gain _total, and the unit becomes a suffix (ms turns into milliseconds). If autocomplete shows a slightly different name, your setup uses another translation setting, so use what it shows. Logs (Loki) Choose Loki and run: {service_name="orders-api"} Your "Processing order" and "Payment declined" lines show up. Click one to expand it, and look for its trace ID. Follow one failure across all three This is the payoff. Try it: 1. In Prometheus, spot the declined rate. 2. In Tempo, open a trace with a red charge_card span. 3. Jump from that span to its logs, and read exactly what happened. The image's Grafana comes pre-wired so you can click from a trace to its logs, and from a log line back to its trace. If a link is missing in your version, query Loki directly. With Loki's native OTLP ingestion, the service name is an index label and the trace ID is structured metadata. So you can filter on the ID you copied from Tempo: {service_name="orders-api"} | trace_id="PASTE_THE_TRACE_ID_HERE" That's one request, seen through all three signals. Why your metrics showed up later than your traces Each signal is batched and sent on its own timer: | Signal | Default delay between exports | Variable | | --- | --- | --- | | Logs | 1 second | OTEL_BLRP_SCHEDULE_DELAY | | Traces | 5 seconds | OTEL_BSP_SCHEDULE_DELAY | | Metrics | 60 seconds | OTEL_METRIC_EXPORT_INTERVAL | Without the 5-second setting we used, you'd wait up to a minute to see your first metric. It's a classic first-day surprise. Do you need your own Collector? In this setup, the Collector inside the Grafana image does the job. You'll want your own as you grow. The OpenTelemetry docs give a few reasons. It gives many services one shared place to send data. It can gather traces across services running on different hosts. And it's a central spot to process data before it reaches a backend. It also keeps backend details out of your apps, so you can switch backends without touching your code. Their advice is that unless you have a single service or you're just experimenting, you'll want a Collector in production. A handy debugging trick When data isn't arriving, run a bare Collector that just prints what it receives. Save this as otel-collector-config.yaml: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 exporters: debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] exporters: [debug] metrics: receivers: [otlp] exporters: [debug] logs: receivers: [otlp] exporters: [debug] Stop the Grafana container first, since both want the same ports. Then run: docker run -p 4317:4317 -p 4318:4318 \ -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \ otel/opentelemetry-collector:latest \ --config=/etc/otel-collector-config.yaml On Windows, use the full path to the file instead of $(pwd). Restart your app and send traffic. Every span, metric and log shows up in the Collector's output. If you see them there, your app is fine and the problem is further downstream. Common first-day problems Most of these are explained above, so this is just an index: | Symptom | Where to look | | --- | --- | | No data anywhere, and no errors | The app wasn't started through opentelemetry-instrument, or the endpoint is wrong. Run the debug Collector above to check. | | Metrics are missing at first | The export delay table above | | INFO logs are missing | The logging gotcha in Step 3 | | Logs vanish with --workers, or everything vanishes with --reload | The callout in Step 4 | | Everything disappeared | The note in Step 1 about where the image keeps its data | From demo to production The demo is deliberately simple. A real setup adds a few things: - A real backend. Pick a managed service or a properly run self-hosted stack in place of the demo image. - Sampling. You usually don't keep every trace. Setting OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 keeps about 10%. - Care with sensitive data. Don't put passwords, tokens or personal data into attributes or log messages. The OpenTelemetry docs have a page on handling sensitive data. - Consistent names. OpenTelemetry has semantic conventions, a shared vocabulary for attribute names like http.route. Following them keeps your data readable across tools. And when one service calls another over HTTP, the instrumentation passes the trace context along in headers. Both services then show up in one trace. That's context propagation, and it's the feature that makes tracing really shine across a system. Where to go next - The OpenTelemetry Demo. A multi-service shop with telemetry already wired in. It's the best place to see tracing across services. - Manual instrumentation for Python. More on spans, events, links and metric instruments. - Collector processors. Filter, transform and sample data before it leaves. - More libraries. Add httpx, SQLAlchemy or a database driver, then rerun opentelemetry-bootstrap -a install. It picks up new libraries on its own. Recap - OpenTelemetry produces telemetry. A backend stores and shows it. - Traces show one request's journey, metrics show trends, logs tell the story. Their power comes from being linked. - opentelemetry-instrument gives you request spans and log shipping for free. You add spans and metrics for your own business logic. - Set OTEL_SERVICE_NAME, and shorten the metric interval while you're learning. - Keep metric attributes low-cardinality. Put the detail on spans. - The otel-lgtm image is a great sandbox, not a production backend. Instrument, export, then follow one request across all three signals. Everything else builds on that. - * * _Details checked against the OpenTelemetry documentation and the grafana/docker-otel-lgtm project on October 25, 2026. Tooling moves quickly, so if a command or menu behaves differently on your version, the official docs win._ Sources - OpenTelemetry Python: Getting Started by Example - OpenTelemetry Python: status and releases - Python logs auto-instrumentation example - Troubleshooting Python automatic instrumentation - OpenTelemetry SDK environment variables - grafana/docker-otel-lgtm on GitHub - Grafana: An OpenTelemetry backend in a Docker image --- Published via ZyVOP — Write once in Markdown, auto-backup to GitHub, and syndicate to Dev.to, Medium & Hashnode in 1 click.
Dockerのビルドを遅くしたコミットをGrafana Tempoのトレースで見つける ! 2026年10月20日に確認した内容です。動作確認には次のバージョンを使いました。 Docker Engine 29.0.0、buildx v0.29.1（Ubuntu 26.04.1） BuildKit v0.33.1（builderのイメージをmoby/buildkit:v0.33.1に固定） grafana/otel-lgtm 0.37.0（Grafana 13.2.3、Tempo 3.1.0を含む） mcp-grafana 2.0.2 Claude Code 2.1.303 gcx 0.3.0 記事中のコマンドは、手元で使ったポート番号をotel-lgtmのデフォルトに戻して書いています。 はじめに こんにちは、Grafana Labsでデベロッパーアドボケイトをしているものです。 2026年11月3日にシンガポールで開かれるGrafana & Friends Singapore x Dockerで話すことになりました。Dockerとの共催というせっかくの機会なので、題材には以前から話そうと思っていたDockerのビルドのトレースを選んでいます。Docker自身もビルドのトレースを画面に出せますが、Grafana Tempoに送って溜めておくと、ビルドをまたいだ調査が可能です。この記事は、その手順と、溜めたトレースをAIに調べさせた結果のまとめです。 TL;DR Dockerのビルドを実行するBuildKitは、ビルドごとにOpenTelemetryのトレースを出しています。そのトレースはdocker buildx history traceでJaeger UIに開けるので、ビルドを1回調べるならその画面で十分です。ただし、BuildKitが残す履歴はデフォルトで48時間か50回分までです。同じトレースをGrafana Tempoに送ってコミットのIDを付けておけば、どのコミットからビルドが遅くなったのかをグラフで特定できます。コーディングエージェントやGrafana Assistantにトレースを読ませれば、遅くしたコミットと手順を見つけるところまで任せられます。 BuildKitが出すトレース docker buildの実体は、buildxというCLIのプラグインと、ビルドを実行するデーモンのBuildKitです。BuildKitはOpenTelemetryでトレースを出す機能を持っていて、Dockerfileの手順ひとつひとつがスパンになります。計装を足す必要はありません。 Dockerの公式ドキュメントが示しているのは、環境変数JAEGER_TRACEでJaegerへ送る方法だけです。一方、BuildKitのREADMEには、OTLPで送る方法が載っています。環境変数OTEL_TRACES_EXPORTER=otlpとOTEL_EXPORTER_OTLP_TRACES_ENDPOINTを使う方法で、後でTempoへ送るときにもこちらを使います。 トレースをどこにも送っていなくても、BuildKitはビルドの履歴にトレースを残しています。docker buildx history traceは、その履歴からトレースを取り出し、一時的に立ち上げたJaeger UIで開くコマンドです1。 docker buildx history trace --builder gf-demo --addr 127.0.0.1:16686 docker buildx history traceで開いたJaeger UIの画面。上部はトレースの概要で、下はDockerfileの手順のスパンのあたりを切り出した この画面では、全体の20.27秒のうち19.24秒を[build 4/4] RUN go mod download && go buildの手順が占めていることがわかります。ビルドが遅いと感じたときに、どの手順を疑えばよいかは、この画面だけで判断できます。 Tempoへ送信 同じトレースをGrafana Tempoへ送るには、buildxのbuilderを作るときに環境変数を2つ渡すだけです。送り先にはgrafana/otel-lgtmを使いました。GrafanaとTempoに加えてPrometheusとOpenTelemetry Collectorも1つにまとめたコンテナで、そのOTLPのポートへ送ります。 docker run -d --name lgtm -p 3000:3000 -p 4317:4317 -p 4318:4318 grafana/otel-lgtm:0.37.0 docker buildx create --name gf-demo --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt network=host \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=127.0.0.1 \ --bootstrap docker buildx build --builder gf-demo --load . docker-containerドライバーのbuilderでは、BuildKitはコンテナの中で動きます。network=hostを付けたのは、そのBuildKitからホストのループバックアドレスに公開したOTLPのポートへ届かせるためです2。 トレースには、buildx（サービス名buildx）とBuildKit（サービス名buildkitd）のスパンが1本につながって入ります。ルートのスパンはbuild .です3。クライアント側のbuildxには環境変数を渡していませんが、buildxのスパンもTempoに届きます4。 同じ種類のビルドのトレースをGrafanaのExploreで開いた画面。上部はトレースの概要で、下はDockerfileの手順のスパンのあたりを切り出した ビルドを遅くしてみる ここまでなら、Dockerの画面で十分です。Tempoへ送る価値が出てくるのは、ビルドが1回ではなく、CI/CDのように継続的に多数行われるときです。 BuildKitの設定では、ビルドの履歴はデフォルトでmaxAge = 172800（48時間）、maxEntries = 50です。docker buildx history traceで開けるのは、この履歴に残っているビルドだけで、--compareで2回分を並べて比べる場合も同じ制約がかかります。 使い捨てのCIのrunnerでは、ジョブが終わればbuilderごと履歴も消えます。DockerはGitHub Actions向けに、ビルドの記録をZIPのアーティファクトとして残す機能を持っています。ただ、それをDocker Desktopに取り込んで見るのは、CIの実行1回分ずつです。「どのコミットからビルドが遅くなったのか」を答えるには、何十回分ものビルドを取得して比較しなければなりません。 そこで、コミットを重ねながら何度もビルドし、全部のトレースをTempoに溜めました。題材は、Prometheusのクライアントライブラリに依存する小さなGoのHTTPサーバーです。はじめは、依存のダウンロードとビルドのキャッシュをキャッシュマウントで再利用するDockerfileでビルドしました。 FROM golang:1.26-alpine AS build WORKDIR /src COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod \ go mod download COPY . . RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build -o /out/hello . FROM gcr.io/distroless/static-debian13 COPY --from=build /out/hello /hello ENTRYPOINT ["/hello"] 途中で、Dockerfileを「整理した」コミットを入れて、キャッシュマウントをわざと外しました。あわせてCOPY . .も依存のダウンロードより前に移した状態です。この順序だと、ソースを1行変えるたびに依存のダウンロードと全パッケージのコンパイルがやり直しになります。buildのステージは以下のように変わっています。 FROM golang:1.26-alpine AS build WORKDIR /src COPY . . RUN go mod download && go build -o /out/hello . 前後には、Dockerfileにラベルを足すだけのコミットを1つずつ混ぜました。git logにはDockerfileを変えたコミットが3つ並ぶので、どれでビルドが遅くなったのかはトレースを見ないとわかりません。 各ビルドには、そのときのコミットのIDを付けました。buildxはOTEL_RESOURCE_ATTRIBUTESを読むので、ここにリソース属性vcs.ref.head.revisionとして入れて渡せば済みます。ただし、この属性が付くのはbuildxのスパンだけで、buildkitdのスパンは対象外です。 OTEL_RESOURCE_ATTRIBUTES="vcs.ref.head.revision=$(git rev-parse --short HEAD)" \ docker buildx build --builder gf-demo . ここまでの各ビルドの所要時間を、TraceQLのメトリクスのクエリでグラフにしました。ルートのスパンだけを選んで、30秒ごとの最大値を取っています5。 { resource.service.name = "buildx" && span:parentID = "" } | max_over_time(duration) 各ビルドの所要時間をTraceQLのmax_over_timeでGrafanaに描いた画面。点の時刻は30秒の区間の終わりを表す トレースで見ると、キャッシュマウントを外す前のビルドは1.9〜2.5秒で終わり、外した後は20.3〜21.7秒に延びて戻りません6。跳ねた直後のトレースを開けば、vcs.ref.head.revisionからコミットがわかります。このデモでは「整理した」コミットでした。遅くなる前後のトレースを並べると、どの手順が延びたのかもわかります。 遅くなる前のビルドのトレースをGrafanaのExploreで開いた画面。依存のダウンロードはキャッシュから読まれ、go buildは約0.9秒で終わっている 遅くなる前は、依存のダウンロードの手順がload cache:で始まるスパンになり、キャッシュから読まれています。遅くなった後は、手順がRUN go mod download && go buildの1つにまとまり、その手順だけで19.36秒かかっています。 手順のスパン名には、RUNのコマンドがそのまま入ります。Dockerfileを変えるとスパン名も変わるので、変更の前後を1本の系列で比べるなら、ルートのスパンの所要時間を使ってください。 エージェントに修正させる Tempoに溜めたトレースは、人だけでなくプログラムからも問い合わせられます。mcp-grafanaは、GrafanaをMCPのサーバーとして公開するツールです。Tempo向けには次のようなツールを持っています。 search_tempo_traces：TraceQLでトレースを検索する query_tempo_metrics：TraceQLのメトリクスを計算する diff_tempo_traces：2本のトレースを比べる これをコーディングエージェントにつなぎ、ビルドが遅くなった原因を調べさせました。mcp-grafanaの起動方法とMCPのクライアントへの登録方法は、mcp-grafanaの公式ドキュメントを参照してください。書き込み系のツールは--disable-writeで外せるので、調査に使うだけなら外しておくと安心です。 今回はClaude Codeを使いましたが、MCPのサーバーを使えるほかのコーディングエージェント（Codexなど）でも同じようにできるでしょう。Gitの履歴の参照やdocker buildx buildの実行など、必要なツールは適宜許可してください。頼んだ内容は次のとおりです。 コンテナのビルドが最近遅くなりました。ビルドのトレースはTempo（Grafanaのデータソース UID "tempo"）に送っています。各ビルドにはサービス buildx の "build ." という名前のスパンがあり、そのスパンのリソース属性 vcs.ref.head.revision にビルドしたコミットが入っています。Tempoのトレースを使って、どのコミットでなぜビルドが遅くなったのかを調べ、このディレクトリのDockerfileを直してください。直したら docker buildx build --builder gf-demo --output type=cacheonly . で再ビルドし、新しいビルドが速くなったことをTempoで確かめてください。新しいトレースが検索できるようになるまで1〜2分かかることがあります。 エージェントは、まずGitの履歴を読み、search_tempo_tracesでコミットごとの所要時間を並べました。そこから導いた結論は、所要時間が約2秒から約21秒に跳ねたのは「整理した」コミットだというものです。遅くなる前と後のトレースをdiff_tempo_tracesで比べ、RUN go mod download && go buildの手順に約19.9秒かかるようになったことと、以前は依存のダウンロードがキャッシュから読まれていたことも確かめました。 修正では、go.modとgo.sumを先にコピーする順序とキャッシュマウントを元に戻し、後から足されたラベルは残しました。再ビルドのあとは新しいトレースが検索できるようになるのを待ち、所要時間が2.31秒に戻ったことをTempoで確かめています。全体で約3分かかり、そのうち約1分半は新しいトレースを待つ時間でした。 オブザーバビリティのSaaSに送る利点 ここまでは、手元のotel-lgtmにトレースを送ってきました。otel-lgtmは、イメージの説明にあるとおり開発、デモ、テストのための環境で、コンテナを消せばトレースも消えます。ビルドをまたいで問いを立てるには、トレースを長く残せて、CIのrunnerからも送れる場所が必要です。Grafana CloudのようなオブザーバビリティのSaaSに送ると、その場所を自分で運用せずに済みます。たとえばGrafana Cloudでも、BuildKitで保持する量よりも圧倒的に多くのトレースを保持できます7。 Grafana Cloudへ送るときも、builderに渡す環境変数を変えるだけです。Grafana CloudのOTLPのゲートウェイにはOTLP/HTTPで送るので、BuildKitのデフォルトのgRPCからhttp/protobufに切り替えます。エンドポイントと認証のヘッダーは、Grafana CloudのポータルのOpenTelemetryのタイルで生成した値をそのまま使いました。 docker buildx create --name gf-cloud --driver docker-container \ --driver-opt image=moby/buildkit:v0.33.1 \ --driver-opt env.OTEL_TRACES_EXPORTER=otlp \ --driver-opt env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces" \ --driver-opt "env.OTEL_EXPORTER_OTLP_TRACES_HEADERS=${OTEL_EXPORTER_OTLP_HEADERS}" \ --bootstrap トレース専用の変数（名前にTRACESが入るもの）だけを渡しているのは、BuildKitがOTEL_EXPORTER_OTLP_ENDPOINTを見るとメトリクスの送信も始めるためです8。このbuilderで手元と同じ20回のビルドを流すと、21本（動作確認の1回を含む）のトレースが全部Grafana CloudのTempoに届きました9。トークンはbuilderのコンテナの環境変数とbuildxの設定ファイルに残るので、使い終わったらdocker buildx rm gf-cloudでbuilderを消してください。 Grafana Assistantに調査させる SaaSの側には、溜めたトレースを調べる機能も最初から用意されています。その例が、Grafana Cloudに組み込まれたAIアシスタントのGrafana Assistantです。ドキュメントによると、AssistantはTempoを含むデータソースに問い合わせ、TraceQLのクエリを書いて実行できます。トレースのIDを渡せば1本のトレースをウォーターフォールで表示でき、実験的な機能として、あるトレースと比べる候補のトレースを探すこともできます。 前の節でエージェントに調べさせたときは、mcp-grafanaを起動してエージェントに登録する準備が必要でした。Assistantは最初からスタックのデータソースにつながっているので、この準備は要らないのが利点です。ビルドが遅くなったことに気づいた人が、Grafanaの画面からそのまま「ビルドの所要時間はいつから延びたか」と聞けば済みます。原因の調査まではAssistantに任せ、Dockerfileを直して再ビルドするところはリポジトリを編集できるコーディングエージェントに任せる、という分担です。 実際に、Grafana Cloudへ送った20回分のビルドについてAssistantに聞きました。AssistantはGrafanaの画面のほか、Grafanaのコマンドラインツールのgcxからも呼べるので、今回はgcxを使いました。伝えたのは、コーディングエージェントに渡したのと同じ前提だけです。 gcx assistant prompt 'コンテナのビルドが最近遅くなりました。ビルドのトレースはTempo（データソース grafanacloud-shophub-traces）に送っています。各ビルドにはサービス buildx の "build ." という名前のスパンがあり、そのスパンのリソース属性 vcs.ref.head.revision にビルドしたコミットが入っています。過去1時間のトレースから、ビルドの所要時間がいつから延びたのか、どのコミットからなのか、ビルドのどの手順が遅くなったのかを調べてください。' Assistantはデータソースへの問い合わせを17回行い、132秒で答えを返しました。答えの冒頭は次のとおりです。 結論: ビルド時間は、およそ 07:09 UTC に約2秒から約20秒へ延びました。最初に遅くなったコミットは 9eac8b9 です。遅くなった手順は Dockerfile の RUN go mod download && go build -o /out/hello . です。 9eac8b9は「整理した」コミットで、答えは正しいものでした。手順についても、速いビルドの6ステップと遅いビルドの4ステップを比べた結果、遅いほうではRUN go mod download && go buildの1つの手順が約19.4秒、全体の約95%を占めるという答えです。依存の取得のキャッシュが効かなくなったことを原因として推測しつつ、「このデータにはコミットの親子順がありません」と書き、時間の順で最初に遅くなったコミットだという限定も付けていました。動作確認のために送った30.2秒のビルドも、コミットの属性がないことに気づいて、ほかのビルドとは別に扱っていました。 Assistantは、自前で運用しているGrafanaからGrafana CloudのAssistantにつないで使うこともできます10。 おわりに Dockerは、ビルドのトレースを出す仕組みも、それを確認する画面も持っています。ビルドを1回調べるならDockerの画面で十分で、Tempoに送る意味が出てくるのはビルドをまたいだ問いを立てるときだと確認しました。 この記事では、長くなるので、ビルドのトレースをGrafanaに送り、溜めたトレースをAIに調べさせるところまでにしました。次は、これを踏まえてCIの設定も含めて書きたいと考えています。CIのジョブがTRACEPARENTを渡せば、ビルドのスパンをジョブのスパンの下に入れて、パイプライン全体の中でビルドを見られるはずです。 ビルドが遅くなったと感じたことがあるなら、まず手元のbuilderに環境変数を2つ足して、トレースを溜めるところから始めてみてください。コンテナビルドでもオブザーバビリティを高めていきましょう。 脚注 標準出力が端末でないときは、Jaeger UIを起動せずに、トレースをJaegerの形式のJSONとして標準出力に書き出しました。 ↩︎ 確認したのはLinuxのDocker Engineだけです。Docker Desktopではnetwork=hostの意味が変わるので、この設定のままでは届かない可能性があります。 ↩︎ buildxは、環境変数TRACEPARENTがあると、それを親のトレースコンテキストとして扱います。この変数が設定されたシェルからビルドすると、ビルドがその親のトレースに入り、build .がルートになりません。ビルドを1回ずつ別のトレースにしたいなら、TRACEPARENTを外してから実行してください。buildx v0.29.1は、cmd/buildx/tracing.goでBuildKit v0.25.0のutil/tracing/envを読み込んでいます。このパッケージがTRACEPARENTとTRACESTATEを読みます。buildxのもっと新しいバージョンでは、同じ役割のパッケージがutil/tracing/childprocessに変わっています。 ↩︎ buildx v0.29.1は、BuildKitに接続するときにclient.WithTracerDelegateを指定しています（driver/manager.go）。buildxは自分のスパンをBuildKitに渡し、BuildKitがまとめて送ります。BuildKitのREADMEにも、buildctlはスパンをbuildkitdに転送するので、exporterはデーモン側だけに設定すればよいとあります。また、docker-containerドライバーではbuildctl dial-stdioというトレースがビルドとは別に毎回届くので、検索するときはresource.service.name = "buildx"で絞ると見やすくなります。 ↩︎ quantile_over_time(duration, .5)で中央値を取ると、約20秒のビルドが、区間を60秒にすると24.3秒、30秒にすると34.4秒になります。quantile_over_timeは所要時間を2のべき乗の幅のバケットに入れて近似するので、値がバケットの境目に寄ります。max_over_timeは、区間に入ったトレースの所要時間の最大値をそのまま返します。 ↩︎ グラフの範囲から外した最初のビルドは、キャッシュが空だったので38.1秒かかりました。 ↩︎ Grafana Cloudの料金のページによると、Tracesの保持期間はFreeプランで14日、Proプランで30日です。BuildKitの履歴の48時間か50回分より長く、手元の開発機のビルドとCIのビルドを同じTempoに入れて、同じクエリで比べられます。 ↩︎ BuildKit v0.33.1のutil/tracing/detect/otlp.goは、OTLPのクライアントをオプションなしで作っています。OpenTelemetryのGoのexporterは、送り先、プロトコル、ヘッダーを環境変数から読みます。 ↩︎ Grafana CloudのTempoでビルドのトレースを検索するときは、name = "build ."で絞ると確実に見つかります。 ↩︎ Grafana Assistantの紹介のページによると、自前で運用しているGrafanaから使う場合も、チャット、ダッシュボードの作成、クエリ、ナビゲーションは使えます。ただし、複数のデータソースを調べてレポートにまとめるInvestigationsと、Grafana CloudのMCPへの接続は、Grafana Cloudに限られた機能です。 ↩︎
【Vol.1】Python特化のOpenTelemetry＋LGTMスタック最小構成 1. はじめに OpenTelemetryの学習にあたり、公式デモを、ようやく動かせるようになったので、それを忘れないうちに、Python（Flask/Locust）特化のシンプルな最小構成環境を作ってみました。 手作業を排除し、EC2上にインフラ構築から自動監視までをTerraformで自動作成します。 :::note info 今回は 【Vol.1】環境構築編 として、まずは動かすことに特化しています。 自分自身の学習記録の備忘録としての意味合いもあります。 ::: !image.png 2. Grafana Labs公式「LGTMスタック」デモとは、 本環境は、Grafana Labs公式の OpenTelemetry Demo をベースにしています。公式デモは、モダンな可観測性（Observability）の標準である 「LGTMスタック」 を手軽に体験できる素晴らしい環境です。 📊 LGTMスタックの構成要素 | コンポーネント | 役割 | | :--- | :--- | | Loki | ログを集約・検索する軽量なログ管理システム | | Grafana | 収集データを1つにまとめて可視化するダッシュボード | | Tempo | リクエストの経路を追跡する分散トレーシング | | Mimir / Prometheus | システム状態（CPUやリクエスト数など）を蓄積するメトリクス | :::note indfo 「LGTMスタック」イメージのみ流用し、アプリ側を 「Python（Flask/Locust）の最小構成」 に絞り込み、ファイル構成、設定内容を理解しやすくしました。 ::: 3. 今回のチャレンジと「あえてやらなかったこと」 今回の最大の目的は、 「必要な部分だけを切り出して、自分なりにカスタマイズしやすい環境を作る」 ことです。 具体的には、以下の要素に絞って手を動かし、理解を深めました。 - Grafanaでの可視化の仕組み - ダッシュボードが「何を、どう表示しているか」の整理 - PromQLや関数を用いたグラフ描画の分析 - OpenTelemetry（メトリクス中心）の挙動確認 - アプリからデータが「どのように出力されるか」の観察 - 負荷（Locust）をかけたときのメトリクス変動の観測 :::note info 「あえてやらなかったこと」は、LGTMスタック各コンポーネント（Loki, Tempo, Mimir, OTel Collector）間の内部接続や詳細設定は公式デモの構成を流用しています。 ::: 4. 技術スタック 本環境で使用している技術スタックは以下の通りです。 - インフラ（IaC）：Terraform - クラウド：AWS (VPC, Subnet, Security Group, EC2 [Amazon Linux 2023]) - コンテナ：Docker / Docker Compose - アプリケーション：Python 3.11 / Flask (Webアプリ) / Locust (負荷テストツール) - 可観測性（Observability）エコシステム： - データ収集：OpenTelemetry - 可視化：Grafana - メトリクス蓄積：Prometheus - ログ蓄積：Loki - 分散トレーシング：Tempo ディレクトリ構成 本環境の全体構成は以下の通りです。 このデータセットは、GitHubリポジトリ に登録しています。 ├── ENV_1011/ # main.tfの構築・プロビジョニング時に使用される環境データ │ ├── my-dashboards/ # Grafanaのダッシュボード設定を格納 │ │ └── grafana-dashboard-python-metrics.json │ ├── python/ # OpenTelemetryによるメトリクス生成を組み込んだアプリ │ │ ├── app.py # Flaskアプリケーション本体 │ │ └── locustfile.py # 負荷試験用のLocustスクリプト │ ├── services/ # EC2上で常時稼働させるsystemd用の設定ファイル │ │ ├── flask-app.service │ │ └── locust-worker.service │ ├── clear-defaults.yaml # 各種環境の初期設定をクリア・調整するための設定 │ ├── my-dashboard-config.yaml # Grafanaダッシュボードのプロバイダー設定 │ └── run-lgtm.sh # LGTMスタック（Grafana等）を起動・制御するスクリプト └── devops_1011/ # 構築・自動化用コード ├── scripts/ │ └── setup.sh # お試し構築・初期セットアップ用スクリプト └── terraform/ # IaC（Infrastructure as Code）関連コード ├── main.tf # AWS EC2インフラやネットワークの定義 └── output.tf # 構築後のパブリックIP等の出力定義 各ディレクトリ・ファイルの役割 >### インフラ自動化・構築 (./devops_1011/) Terraformを用いて、AWSのEC2上にインフラ環境を自動構築します。 !image.png >### アプリケーション & 自動監視環境 ( ./ENV_1011/ ) main.tf を実行し、EC2インスタンス起動後に、環境構築/python実行で必要なファイルを格納しています。 !image.png 5. システム構成 !image.png 今回はGrafana Labs公式デモ版のコンテナイメージ（grafana/otel-lgtm）をベースに使用しているため、OpenTelemetry Collector部や各バックエンド（Prometheus / Tempo / Loki / Pyroscope）の内部構成はそちらに依存しています。 このシステムに対して、自作した「Python Web App (Flask)」と「負荷生成器（Locust）」を接続し、EC2インスタンス（t3.small）内でsystemdとしてバックグラウンド実行させています。 インフラ構築や環境配備の自動化を図るために、Terraform でAWSリソースの自動生成を行い、EC2のUser DataをトリガーにGitHub・Docker Hubと連携してデプロイを自動化しています。 - GitHub：最新のソースコード（app.py / locustfile.py）を自動取得 - Docker Hub：可視化基盤のコンテナイメージ（grafana/otel-lgtm）をプル 🔄 各コンポーネントの役割 !image.png 6. メトリクス ダッシュボードで可視化したメトリクスは以下の通りです。 Locustの負荷変動に応じて、これらの値も変動します。 !image.png 具体的なコードについては、末尾の [【付録：環境構築用コード】](#10付録環境構築用コード) にある「grafana-dashboard-python-metrics.json」を参照してください。 :::note info 今回の構成ではメトリクス収集に特化しており、LokiやTempoへのデータ送信に必要な機能実装は行わなかったので、 ログ、トレース の内容は見られません。 ::: 7.デプロイ手順 インフラのプロビジョニングからアプリのデプロイ、監視の有効化まで、Terraformで自動化しています。 >### 1. 環境変数の設定 ローカル環境に自身のAWSの認証情報（アクセスキーなど）をセットします。 >### 2. Terraformの実行 terraform apply を実行します。AWS上にVPC、セキュリティグループ、EC2（Amazon Linux 2023）が自動生成されます。 ※具体的なIaCコードは末尾の [【付録：環境構築用コード】](#10付録環境構築用コード) を参照。 >### 3. user_dataによる完全自動セットアップ EC2の起動時、user_dataがバックグラウンドで以下の処理を全自動で実行します。 1. GitのSparse Checkout（部分チェックアウト） を使い、リポジトリから必要なディレクトリ（ENV_1011）のみダウンロード。 2. Docker / Docker Compose のインストールと起動。 3. 公式スクリプト（run-lgtm.sh）によるLGTMスタックコンテナ起動。 4. Python 3.11環境の構築と、必要ライブラリ（Flask, OpenTelemetry SDK, Locustなど）のインストール。 5. Pythonスクリプトの配置、およびsystemdへのサービス登録と自動起動。 >### 4. Grafanaへのアクセス terraform apply 完了後、数分待つだけでセットアップが完了します。出力されたパブリックIPの3000番ポート（http://<EC2のパブリックIP>:3000）へアクセスすれば、すぐに完成したダッシュボードを確認できます。 <details> <summary>詳細手順は、こちらを参照</summary> >### URL指定(Grafana-GUI) terraform apply 完了後、以下の様にパブリックIPが出力されます。 ec2_private_ip = "10.0.1.206" ec2_public_ip = "3.112.205.128" 上記のケースでは、以下のURL に2～3分待ってアクセスします。 3.112.205.128 >### ダッシュボード選択 以下の様に、Grafana からダッシュボード選択を行います。 !image.png >### メニュー選択 表示期間を「⏱ Last 5 minutes」に切り替えて、10分ほど待つと、以下のようなデータが見られるようになります。 !image.png </details> 8. おわりに 今回の取り組みを通じ、OpenTelemetryの強力さとLGTMスタックによる監視基盤の構築方法を深く学ぶことができました。 手間の多い手動構築（SSH、クローン、権限変更、サービス登録など）を、Terraformの user_data と Gitの Sparse Checkout で自動化し、「コマンドで環境構築」する面白味を実感できました。 >### 周辺技術での学び 環境コード化にあたり、監視以外の実用的なスキルも習得できました。 - Locustによる動的負荷試験：リクエストを変動させ、意図的にメトリクスの波（トレンド）を作る手法 - Terraformの user_data：EC2起動時のプロビジョニングを完全自動化する実務的な手法 9. 今後の展望 今回は【Vol.1】として「最小構成で動かす」ことを最優先しました。このベース環境を活かし、以下の要素にも挑戦する予定です。(あくまでも予定です) - 監視基盤の自作化：公式デモの流用を脱却し、各コンポーネントとCollector間の接続・設定をイチから自作する - 自動制御の組み込み：各バックエンドのデータをシェルスクリプト等で定期取得し、異常を自動検知する仕組みの実装。 - 構成管理の高度化：user_data（シェルスクリプト）で行っているアプリ配置やサービス登録を、Ansibleによる管理へ移行する。 - オブザーバビリティの完全化：今回はメトリクスのみのだった構成にLoki（ログ）とTempo（トレース）の実装を本格追加し、「3つの柱」を可視化できる環境の構築。 10.付録：環境構築用コード GitHubレポジトリ に登録しています <details> <summary>./terraform/main.tf</summary> terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "ap-northeast-1" } # 1. VPC（仮想ネットワーク）の作成 resource "aws_vpc" "main" { cidr_block = "10.0.0.0/16" enable_dns_hostnames = true tags = { Name = "study-vpc" } } # 2. パブリックサブネットの作成 resource "aws_subnet" "public" { vpc_id = aws_vpc.main.id cidr_block = "10.0.1.0/24" map_public_ip_on_launch = true # 起動時にパブリックIPを自動割り当て tags = { Name = "study-public-subnet" } } # 3. インターネットゲートウェイの作成（外の世界と繋ぐ窓口） resource "aws_internet_gateway" "igw" { vpc_id = aws_vpc.main.id tags = { Name = "study-igw" } } # 4. ルートテーブルの作成と紐付け（ネットへの道を登録） resource "aws_route_table" "public" { vpc_id = aws_vpc.main.id route { cidr_block = "0.0.0.0/0" gateway_id = aws_internet_gateway.igw.id } tags = { Name = "study-public-rt" } } resource "aws_route_table_association" "public" { subnet_id = aws_subnet.public.id route_table_id = aws_route_table.public.id } # ================================================================================= # 1. セキュリティグループの定義 # ================================================================================= resource "aws_security_group" "ec2" { name = "study-ec2-sg" description = "Security Group for OpenTelemetry and App Server" vpc_id = aws_vpc.main.id # ※環境に合わせてVPC IDの変数名を調整してください # ------------------------------------------------------------------------------- # 各種アプリケーション・可観測性ツール用のインバウンドルール（dynamicブロックで一括管理） # ------------------------------------------------------------------------------- dynamic "ingress" { for_each = { 22 = "SSH" 3000 = "Grafana UI" 16686 = "Jaeger UI" 9090 = "Prometheus UI" 3100 = "Loki" 8000 = "Python Web App" 9464 = "OpenTelemetry Prometheus Exporter" 5000 = "Python App (Flask)" 4317 = "OpenTelemetry (OTLP/gRPC)" 4318 = "OpenTelemetry (OTLP/HTTP)" 8080 = "OpenTelemetry Demo Web & Grafana" } content { description = ingress.value from_port = ingress.key to_port = ingress.key protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } } # ------------------------------------------------------------------------------- # 【超重要】EC2同士（VPC内部）の全通信を許可するルール # ------------------------------------------------------------------------------- ingress { description = "VPC Internal All Traffic" from_port = 0 to_port = 0 protocol = "-1" # すべてのプロトコルを許可 cidr_blocks = ["10.0.0.0/16"] # VPC内部のIPアドレスからの通信をすべて許可 } # ------------------------------------------------------------------------------- # インターネットへの外向き通信をすべて許可（重複を排除して1つにスッキリ整理） # ------------------------------------------------------------------------------- egress { description = "Allow all outbound traffic" from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } # ================================================================================= # 2. SSH接続用のキーペアを自動生成 # ================================================================================= resource "tls_private_key" "keygen" { algorithm = "RSA" rsa_bits = 4096 } resource "aws_key_pair" "keypair" { key_name = "study-key" public_key = tls_private_key.keygen.public_key_openssh } # 生成された秘密鍵（Terra Termで使う.pemファイル）をローカルに保存 resource "local_file" "private_key" { content = tls_private_key.keygen.private_key_pem filename = "${path.module}/study-key.pem" } # ================================================================================= # 3. AWSから自動で最新のAmazon Linux 2023 AMI IDを取得 # ================================================================================= data "aws_ssm_parameter" "amzn_linux_2023" { name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64" } # ================================================================================= # 4. EC2インスタンスの作成（GitHubパブリック方式・完全自動化版） # ================================================================================= resource "aws_instance" "app_server" { ami = data.aws_ssm_parameter.amzn_linux_2023.value instance_type = "t3.small" subnet_id = aws_subnet.public.id vpc_security_group_ids = [aws_security_group.ec2.id] key_name = aws_key_pair.keypair.key_name # ルートディスク容量を30GB（無料枠の最大値）に増やす設定 root_block_device { volume_size = 30 volume_type = "gp3" } tags = { Name = "study-ec2-1" } # =============================================================================== # 【完全自動化】初期化シェルスクリプト（user_data） # =============================================================================== user_data = <<-EOF #!/bin/bash # エラーが発生したら即座にスクリプトを終了させる設定 set -euo pipefail echo "==========================================================" echo "=== 1. GitHubから特定のディレクトリ（ENV_1011）のみを取得 ===" echo "==========================================================" dnf install -y git sleep 5 mkdir -p /opt/deploy_data cd /opt/deploy_data # Sparse Checkout（部分チェックアウト）の設定 git init git remote add origin https://github.com/wata123-t/deploy_data git config core.sparseCheckout true echo "ENV_1011/" >> .git/info/sparse-checkout # データの引っ張りと移動 git pull origin main cd ENV_1011 echo "==========================================================" echo "=== 2. Docker & Docker Compose のインストールと起動 ===" echo "==========================================================" dnf install -y docker sleep 5 systemctl enable --now docker # Docker Compose V2 のインストール mkdir -p /usr/local/lib/docker/cli-plugins/ curl -SL https://github.com/docker/compose/releases/download/v2.29.2/docker-compose-linux-x86_64 -o /usr/local/lib/docker/cli-plugins/docker-compose chmod +x /usr/local/lib/docker/cli-plugins/docker-compose echo "==========================================================" echo "=== 3. Grafana Labs公式「LGTMスタック」デモの実行 ===" echo "==========================================================" # すでにGit側で権限を付与していますが、念のため再付与して実行 chmod +x ./run-lgtm.sh ./run-lgtm.sh sleep 10 echo "==========================================================" echo "=== 4. Python 3.11 & 開発ツールのインストール ===" echo "==========================================================" dnf install -y python3.11 python3.11-pip python3.11-devel gcc sleep 10 echo "==========================================================" echo "=== 5. 必要なPythonライブラリの一括インストール ===" echo "==========================================================" python3.11 -m pip install --upgrade pip sleep 10 python3.11 -m pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-system-metrics locust sleep 10 echo "==========================================================" echo "=== 6. アプリケーションフォルダの作成とファイルのコピー ===" echo "==========================================================" mkdir -p /opt/script/python/app # GitHubから落としてきた相対パスからファイルをコピー cp "./python/app.py" /opt/script/python/app/app.py cp "./python/locustfile.py" /opt/script/python/locustfile.py echo "==========================================================" echo "=== 7. 各種サービス（systemd）ファイルの配置と起動 ===" echo "==========================================================" cp "./services/flask-app.service" /etc/systemd/system/flask-app.service cp "./services/locust-worker.service" /etc/systemd/system/locust-worker.service systemctl daemon-reload systemctl enable --now flask-app locust-worker echo "==========================================================" echo "=== すべてのセットアップ処理が正常に完了しました！ ===" echo "==========================================================" EOF } </details> <details> <summary>./terraform/output.tf</summary> # ================================================================================= # 5. アウトプット（接続用IPアドレスの表示） # ================================================================================= output "ec2_public_ip" { value = aws_instance.app_server.public_ip description = "起動したEC2のパブリックIPアドレスです" } output "ec2_private_ip" { value = aws_instance.app_server.private_ip description = "EC2のプライベートIPアドレスです" } </details> <details> <summary>./my-dashboards/grafana-dashboard-python-metrics.json </summary> { "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "datasource", "uid": "grafana" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "description": "Dashboard for **Python** metrics with OpenTelemetry instrumentation", "editable": true, "fiscalYearStartMonth": 0, "gnetId": 18812, "graphTooltip": 0, "links": [ { "asDropdown": false, "icon": "info", "includeVars": false, "keepTime": false, "tags": [], "targetBlank": false, "title": "Semantic Conventions: 1.20.0 - 1.22.0", "tooltip": "multiple versions of the semantic conventions are supported using 'or' and regex queries", "type": "link", "url": "https://github.com/open-telemetry/semantic-conventions/blob/main/schemas/1.20.0" } ], "liveNow": false, "panels": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "HTTP server request rate", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "reqps" }, "overrides": [] }, "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 }, "id": 51, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "sum by (instance) (rate({__name__=~\"http_server_request_duration_seconds_count|http_server_request_duration_count|http_server_duration_seconds_count|http_server_duration_count|http_server_duration_milliseconds_count|http_server_requests_milliseconds_count|http_server_requests_count|http_server_requests_seconds_count\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))", "instant": false, "legendFormat": "__auto", "range": true, "refId": "A" } ], "title": "Rate", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "HTTP server error ratio - ratio of requests that return 5xx", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "percentunit" }, "overrides": [] }, "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 }, "id": 50, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "(sum(rate(http_server_duration_milliseconds_count{http_status_code=~\"5..\"}[$__rate_interval])) / sum(rate(http_server_duration_milliseconds_count[$__rate_interval]))*1) or on() vector(0)", "instant": false, "legendFormat": "__auto", "range": true, "refId": "A" } ], "title": "Error %", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "95th percentile of HTTP server request duration in seconds", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "s" }, "overrides": [] }, "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 }, "id": 52, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "(histogram_quantile(0.95, sum by(le, instance) (rate({__name__=~\"http_server_request_duration_seconds_bucket|http_server_request_duration_bucket|http_server_duration_seconds_bucket|http_server_duration_bucket|http_server_requests_seconds_bucket\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))))\nor\n(histogram_quantile(0.95, sum by(le, instance) (rate({__name__=~\"http_server_duration_milliseconds_bucket|http_server_requests_milliseconds_bucket|http_server_requests_bucket\", job=~\"$job\", instance=~\"$instance\"}[$__rate_interval]))) / 1000)\n", "instant": false, "legendFormat": "{{instance}}", "range": true, "refId": "A" } ], "title": "Duration (95%)", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "CPU utilization for the whole system", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "percentunit" }, "overrides": [] }, "gridPos": { "h": 7, "w": 13, "x": 0, "y": 8 }, "id": 38, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "process_cpu_utilization_ratio", "legendFormat": "{{instance}}", "range": true, "refId": "A" } ], "title": "CPU utilization", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "Used heap memory / heap memory limit ", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "bytes" }, "overrides": [] }, "gridPos": { "h": 7, "w": 11, "x": 13, "y": 8 }, "id": 30, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "process_memory_usage_bytes", "legendFormat": "{{instance}}", "range": true, "refId": "A" } ], "title": "Heap Memory utilization", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "Percentage of time spend for garbage collection pauses", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bar", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 7, "w": 13, "x": 0, "y": 15 }, "id": 46, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "sum by (instance) (rate(cpython_gc_collections_total[$__rate_interval]))*60", "hide": false, "legendFormat": "__auto", "range": true, "refId": "B" } ], "title": "Garbage Collection", "type": "timeseries" }, { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "description": "Number of currently executing (also called \"live\") threads", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "none" }, "overrides": [] }, "gridPos": { "h": 7, "w": 11, "x": 13, "y": 15 }, "id": 42, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "prometheus", "uid": "${datasource}" }, "editorMode": "code", "expr": "process_thread_count", "legendFormat": "{{instance}}", "range": true, "refId": "A" } ], "title": "Threads", "type": "timeseries" } ], "refresh": "", "schemaVersion": 38, "tags": ["Python"], "templating": { "list": [ { "current": { "selected": false, "text": "Prometheus", "value": "prometheus" }, "description": "Choose a Prometheus data source", "hide": 0, "includeAll": false, "label": "Data source", "multi": false, "name": "datasource", "options": [], "query": "prometheus", "queryValue": "", "refresh": 1, "regex": "", "skipUrlSync": false, "type": "datasource" }, { "allValue": ".+", "current": { "selected": false, "text": "All", "value": "$__all" }, "datasource": { "type": "prometheus", "uid": "${datasource}" }, "definition": "label_values(http_server_request_duration_seconds_count, job)", "hide": 0, "includeAll": true, "label": "Job", "multi": true, "name": "job", "options": [], "query": { "query": "label_values(http_server_request_duration_seconds_count, job)", "refId": "PrometheusVariableQueryEditor-VariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 1, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "allValue": ".+", "current": { "selected": false, "text": "All", "value": "$__all" }, "datasource": { "type": "prometheus", "uid": "${datasource}" }, "definition": "label_values(http_server_request_duration_seconds_count, instance)", "description": "The instance of the application, e.g. pod1", "hide": 0, "includeAll": true, "label": "Instance", "multi": true, "name": "instance", "options": [], "query": { "query": "label_values(http_server_request_duration_seconds_count, instance)", "refId": "PrometheusVariableQueryEditor-VariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 1, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false } ] }, "time": { "from": "now-1h", "to": "now" }, "timepicker": { "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d"] }, "timezone": "", "title": "python-Objability", "uid": "python-dashboard-metrics", "version": 1, "weekStart": "" } </details>