You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "JackieTien97 (via GitHub)" <gi...@apache.org> on 2026/04/18 06:39:13 UTC
[PR] Add OpenTelemetry (OTLP/HTTP) receiver for AI coding assistant telemetry [iotdb]
JackieTien97 opened a new pull request, #17516:
URL: https://github.com/apache/iotdb/pull/17516
## Summary
This PR adds an **OTLP/HTTP receiver** to IoTDB's REST service, enabling IoTDB to directly ingest [OpenTelemetry](https://opentelemetry.io/) telemetry data (traces, metrics, and logs) from AI coding assistants such as **Claude Code**, **Codex**, **Gemini CLI**, and others.
### Motivation
Modern AI coding assistants like Claude Code emit rich telemetry data via the OpenTelemetry protocol — including token usage, API costs, tool execution stats, and session events. Storing this data locally in IoTDB enables developers and teams to:
- Analyze token consumption patterns across models (input/output/cacheRead/cacheCreation)
- Track API costs per session and over time
- Monitor tool execution success rates and latencies
- Correlate user prompts with API calls and tool invocations via `prompt.id`
### How It Works
The OTLP receiver is embedded in the existing REST service (port 18080) and exposes three standard OTLP/HTTP endpoints:
```
POST /rest/v1/otlp/v1/traces
POST /rest/v1/otlp/v1/metrics
POST /rest/v1/otlp/v1/logs
```
Both `application/x-protobuf` and `application/json` content types are supported.
**Dynamic database routing**: The receiver derives the target database name from each request's `service.name` resource attribute. For example:
- `claude-code` → database `claude_code`
- `codex` → database `codex`
- `Gemini CLI` → database `gemini_cli`
Database and tables (`metrics`, `logs`, `traces`) are created automatically on first use.
### Table Schema Design (Table Model)
**`metrics` table** — one row per data point, with OTLP attributes extracted into typed columns:
| Column | Category | Description |
|--------|----------|-------------|
| `user_id` | TAG | Anonymous device identifier |
| `session_id` | TAG | Session UUID |
| `metric_name` | TAG | e.g. `claude_code.token.usage`, `claude_code.cost.usage` |
| `model` | TAG | Model name, e.g. `claude-opus-4-7` |
| `type` | TAG | Metric sub-type: `input`/`output`/`cacheRead`/`cacheCreation`/`user`/`cli` |
| `terminal_type` | ATTRIBUTE | `iTerm.app` / `vscode` / `cursor` |
| `service_version` | ATTRIBUTE | Claude Code version |
| `os_type`, `os_version`, `host_arch` | ATTRIBUTE | Platform info |
| `unit`, `metric_type`, `description` | ATTRIBUTE | Metric metadata |
| `value` | FIELD | The metric value (tokens count / USD / seconds) |
**`logs` table** — flattened event records with event-type-specific columns:
| Column | Category | Description |
|--------|----------|-------------|
| `user_id`, `session_id` | TAG | Identity |
| `event_name` | TAG | `user_prompt` / `api_request` / `api_error` / `tool_result` / `tool_decision` |
| `terminal_type`, `service_version`, `os_type`, `host_arch` | ATTRIBUTE | Static per-session |
| `prompt_id`, `event_sequence`, `body` | FIELD | Common event fields |
| `model`, `cost_usd`, `duration_ms`, `input_tokens`, `output_tokens`, ... | FIELD | API request fields |
| `tool_name`, `success`, `tool_duration_ms`, `decision`, `decision_source` | FIELD | Tool fields |
**`traces` table** — standard span data with resource attributes extracted.
### How to Use with Claude Code
1. Enable REST service in `iotdb-system.properties`:
```properties
enable_rest_service=true
rest_service_port=18080
```
2. Add the following environment variables to your shell profile (e.g. `~/.zshrc` or `~/.bashrc`):
```bash
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:18080/rest/v1/otlp
```
3. Start IoTDB, then start a new Claude Code session. Telemetry data will flow automatically.
4. Query your data:
```sql
-- Token usage by model and type
USE claude_code;
SELECT model, type, sum(value) FROM metrics
WHERE metric_name = 'claude_code.token.usage'
GROUP BY model, type;
-- API cost trend
SELECT date_bin(1h, time) AS hour, sum(value) AS cost
FROM metrics WHERE metric_name = 'claude_code.cost.usage'
GROUP BY 1 ORDER BY 1;
-- Tool success rate
SELECT tool_name, count_if(success = 'true') AS ok, count(*) AS total
FROM logs WHERE event_name = 'tool_result'
GROUP BY tool_name;
-- Trace a single prompt
SELECT * FROM logs WHERE prompt_id = 'xxx' ORDER BY event_sequence;
```
### Files Changed
**New files** (11 files in `external-service-impl/rest/.../protocol/otlp/v1/`):
- `OtlpTracesResource`, `OtlpMetricsResource`, `OtlpLogsResource` — JAX-RS endpoints
- `OtlpService` — per-database session management + schema init
- `OtlpSchemaInitializer` — idempotent `CREATE DATABASE/TABLE IF NOT EXISTS`
- `OtlpIngestor` + `OtlpTableBatch` — column-major batch → `InsertTabletStatement`
- `OtlpTracesConverter`, `OtlpMetricsConverter`, `OtlpLogsConverter` — OTLP→rows
- `OtlpConverter` — timestamp precision, hex encoding, attribute extraction
- `OtlpHttp` — protobuf/JSON parsing and response rendering
**Modified files** (5 files):
- `rest/pom.xml` — added `io.opentelemetry.proto:opentelemetry-proto` + `protobuf-java-util`
- `AuthorizationFilter` — bypass auth for `/rest/v1/otlp/` paths (OTLP receiver authenticates internally via `otlp_username/otlp_password`)
- `IoTDBRestServiceConfig/Descriptor` — added `otlp_username`/`otlp_password` config
- `iotdb-system.properties.template` — documented OTLP configuration section
## Test plan
- [x] Build: `mvn clean package -pl distribution -am -DskipTests` succeeds
- [x] Smoke test: `curl -X POST http://localhost:18080/rest/v1/otlp/v1/logs` with JSON body returns HTTP 200
- [x] Verify metrics: token.usage with type=input/output creates separate rows in `claude_code.metrics`
- [x] Verify logs: api_request and tool_result events land with all fields extracted
- [x] Multi-database: requests from different service.name values create separate databases
- [x] End-to-end: real Claude Code telemetry data flows into IoTDB tables
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry (OTLP/HTTP) receiver for AI coding assistant telemetry [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on PR #17516:
URL: https://github.com/apache/iotdb/pull/17516#issuecomment-4627973369
Functional issues I noticed in this PR:
- The PR does not compile against current master. After merging with latest master, the new OTLP classes still reference IClientSession.SqlDialect.TABLE, but IClientSession no longer has the nested SqlDialect enum; current code uses org.apache.iotdb.commons.queryengine.common.SqlDialect. Affected locations include OtlpService.java:97/117, OtlpIngestor.java:69, and OtlpSchemaInitializer.java:145.
- The OTLP endpoint bypasses REST Basic auth and then writes as the configured IoTDB user, which defaults to oot/root. With enable_rest_service=true, any unauthenticated caller to /rest/v1/otlp/** can create databases/tables and insert data as root. See AuthorizationFilter.java:62-69 and the default otlp_username / otlp_password config. This should probably require explicit enablement and authentication/token support, and should not default to root.
- Schema initialization failures are swallowed, but the session is still cached as ready. OtlpSchemaInitializer.runDdl() only logs non-success statuses/exceptions, while OtlpService.openDatabaseSession() caches the session afterwards. If CREATE DATABASE/TABLE fails due to permission, transient errors, or partial DDL failure, later requests for that database will skip schema init forever and keep failing inserts. The initializer should fail the first ingest and avoid caching until all required DDL succeeds.
- Missing OTLP fields are written as concrete defaults instead of nulls. OtlpTableBatch.bitMaps() returns fresh unmarked bitmaps, columnValues() fills missing string slots with empty Binary, and numeric columns keep Java default 0. For logs, many event-specific fields are intentionally not set for other event types, so they become
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry (OTLP/HTTP) receiver for AI coding assistant telemetry [iotdb]
Posted by "Necmttn (via GitHub)" <gi...@apache.org>.
Necmttn commented on PR #17516:
URL: https://github.com/apache/iotdb/pull/17516#issuecomment-4761515110
One compatibility edge worth testing: not every assistant exposes the useful data as spans.
For example, Codex-style assistant telemetry is easiest to consume as OTLP JSON logs through the logs endpoint, with the conversation/session id as the correlation key. Claude-style metrics often correlate through prompt/session attributes instead.
If this receiver is meant to be cross-assistant, I would add fixtures that post JSON logs with Codex-shaped attributes and assert that `service.name`, `conversation.id` or session id, model, event name, and token fields survive as queryable columns/tags. Otherwise the Claude metric path can work while Codex sessions quietly fragment or disappear from dashboards.
---
_Generated with [ax](https://github.com/Necmttn/ax)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add OpenTelemetry (OTLP/HTTP) receiver for AI coding assistant telemetry [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #17516:
URL: https://github.com/apache/iotdb/pull/17516#issuecomment-4273159521
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/17516?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `10 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 39.86%. Comparing base ([`aa4b551`](https://app.codecov.io/gh/apache/iotdb/commit/aa4b551befb9c8fcdb880503e27a8b9f8e1550f8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f2e86e3`](https://app.codecov.io/gh/apache/iotdb/commit/f2e86e3edf082a9cad76fe72b2ae35828a14b97a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/17516?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...che/iotdb/db/conf/rest/IoTDBRestServiceConfig.java](https://app.codecov.io/gh/apache/iotdb/pull/17516?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fconf%2Frest%2FIoTDBRestServiceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvY29uZi9yZXN0L0lvVERCUmVzdFNlcnZpY2VDb25maWcuamF2YQ==) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...iotdb/db/conf/rest/IoTDBRestServiceDescriptor.java](https://app.codecov.io/gh/apache/iotdb/pull/17516?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fconf%2Frest%2FIoTDBRestServiceDescriptor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvY29uZi9yZXN0L0lvVERCUmVzdFNlcnZpY2VEZXNjcmlwdG9yLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17516 +/- ##
============================================
- Coverage 39.87% 39.86% -0.01%
Complexity 312 312
============================================
Files 5137 5137
Lines 347150 347160 +10
Branches 44247 44247
============================================
- Hits 138420 138403 -17
- Misses 208730 208757 +27
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/17516?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org