You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flagon.apache.org by au...@apache.org on 2026/10/12 00:56:42 UTC
(flagon) branch feat/opensearch-and-apache-lakehouse-examples created (now 0eb3cb3)
This is an automated email from the ASF dual-hosted git repository.
brucearctor pushed a change to branch feat/opensearch-and-apache-lakehouse-examples
in repository https://gitbox.apache.org/repos/asf/flagon.git
at 0eb3cb3 feat(examples): add OpenSearch (#54) and All-Apache Lakehouse (Kafka, Iceberg, DataFusion, Distill) examples
This branch includes the following new commits:
new 0eb3cb3 feat(examples): add OpenSearch (#54) and All-Apache Lakehouse (Kafka, Iceberg, DataFusion, Distill) examples
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(flagon) 01/01: feat(examples): add OpenSearch (#54) and All-Apache Lakehouse (Kafka, Iceberg, DataFusion, Distill) examples
Posted by au...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brucearctor pushed a commit to branch feat/opensearch-and-apache-lakehouse-examples
in repository https://gitbox.apache.org/repos/asf/flagon.git
commit 0eb3cb317a820c2bb88fb45c375a6c447b0adfed
Author: Bruce Arctor <50...@users.noreply.github.com>
AuthorDate: Sun Oct 11 17:56:01 2026 -0700
feat(examples): add OpenSearch (#54) and All-Apache Lakehouse (Kafka, Iceberg, DataFusion, Distill) examples
---
examples/README.md | 29 +++
examples/apache-lakehouse/README.md | 90 +++++++++
.../apache-lakehouse/distill_lakehouse_demo.py | 208 +++++++++++++++++++++
examples/apache-lakehouse/docker-compose.yml | 67 +++++++
.../apache-lakehouse/otel-collector-config.yaml | 60 ++++++
examples/opensearch/README.md | 100 ++++++++++
examples/opensearch/docker-compose.yml | 64 +++++++
examples/opensearch/kubernetes/opensearch.yaml | 188 +++++++++++++++++++
examples/opensearch/otel-collector-config.yaml | 68 +++++++
examples/opensearch/userale-index-template.json | 82 ++++++++
10 files changed, 956 insertions(+)
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 0000000..6e90436
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,29 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Apache Flagon Deployment & Analytics Examples
+
+Apache Flagon UserALE.js and Distill support multiple open-standard wire formats (`userale-json`, `otlp-json`, and `proto-json`), allowing you to choose the right backend architecture for your workload:
+
+| Example | Stack & License | Best For |
+| :--- | :--- | :--- |
+| **[OpenTelemetry + Grafana/Loki/Tempo](../products/userale/example/otel-demo/README.md)** | `OtlpJsonExporter` $\rightarrow$ OTel Collector $\rightarrow$ Loki + Tempo + Grafana | **SRE & Observability Teams**: Correlating frontend user interactions, rage-clicks, and errors with distributed backend traces (`traceId` / `spanId`). |
+| **[All-Apache Behavioral Lakehouse](./apache-lakehouse/README.md)** | `OtlpJsonExporter` $\rightarrow$ OTel Collector $\rightarrow$ **Apache Kafka** $\rightarrow$ **Apache Iceberg / Parquet** $\rightarrow$ **Apache DataFusion** + **Apache Flagon Distill** (`100% Apache-2.0`) | **Product Analytics & Data Science at Scale**: Columnar storage without inverted-index bloat, fast SQL window functions (`LEAD`/`LAG` funnels), and zero-copy Python analytics with Distill. |
+| **[OpenSearch Stack](./opensearch/README.md)** | `OtlpJsonExporter` $\rightarrow$ OTel Collector $\rightarrow$ **OpenSearch** + **OpenSearch Dashboards** (`100% Apache-2.0`) | **Search & Log Indexing**: Drop-in `Apache-2.0` open-source replacement for the legacy ELK stack (Docker Compose + Kubernetes). |
+| **[Legacy ELK Stack](./elk/docker/README.md)** | UserALE JSON $\rightarrow$ Logstash $\rightarrow$ Elasticsearch + Kibana | Legacy compatibility for existing Elasticsearch deployments. |
diff --git a/examples/apache-lakehouse/README.md b/examples/apache-lakehouse/README.md
new file mode 100644
index 0000000..dfa6d29
--- /dev/null
+++ b/examples/apache-lakehouse/README.md
@@ -0,0 +1,90 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# The "All-Apache" Behavioral Lakehouse Example
+
+**Apache Flagon UserALE.js + Apache Kafka + Apache Iceberg + Apache DataFusion + Apache Flagon Distill + Apache Superset**
+
+This directory demonstrates a modern, **100% Apache Software Foundation (`Apache-2.0`)** streaming lakehouse and behavioral OLAP architecture for high-volume frontend telemetry.
+
+---
+
+## Why a Behavioral Lakehouse Instead of Legacy ELK?
+
+1. **Zero Inverted-Index Amplification**: High-frequency UserALE interaction streams (`mousemove`/`scroll` intervals, `click` coordinates, `mutation` summaries, and `rrweb` session replay envelopes) are stored in columnar **Apache Parquet / Apache Iceberg** tables rather than expensive JVM heap and Lucene inverted indexes.
+2. **Native SQL Window Functions for Funnels & Sequences**: Vectorized engines like **Apache DataFusion** (and **Apache Pinot** / **DuckDB**) execute ordered session window queries (`LEAD()`, `LAG()`, `OVER (PARTITION BY session_id ORDER BY client_time)`) in milliseconds.
+3. **Zero-Copy Python Analytics with Apache Flagon Distill**: **Distill** reads Arrow/Parquet/Iceberg partitions directly into Python to compute UX frustration scores (`distill.frustration_score`), Markov transition matrices (`distill.transition_matrix`), spatial click heatmaps (`distill.click_heatmap_data`), and OpenTelemetry trace correlations (`distill.correlate_by_trace`).
+4. **100% ASF Category-A Open Source**: Every component in this stack is governed by the Apache Software Foundation (or CNCF) under the permissive **Apache-2.0** license—no BSL 1.1 or AGPLv3 dependencies.
+
+---
+
+## Architecture
+
+```
+Browser (Apache Flagon UserALE.js — OtlpJsonExporter / ProtoJsonExporter)
+ │ POST http://localhost:4318/v1/logs (OTLP/JSON + W3C traceId/spanId)
+ ▼
+OpenTelemetry Collector (otelcol-contrib, Apache-2.0)
+ ├──► Apache Kafka (apache/kafka-native KRaft, Apache-2.0)
+ │ Topic: flagon-userale-events
+ ▼
+Apache Iceberg REST Catalog + Apache Parquet Warehouse (Apache-2.0)
+ ├──► Apache DataFusion / DuckDB SQL (Vectorized Funnels, Session Windows, Rage-Click OLAP)
+ ├──► Apache Flagon Distill (Python Frustration, Spatial, Markov & OTel Trace Analytics)
+ └──► Apache Superset (Interactive BI Dashboards)
+```
+
+---
+
+## Quickstart
+
+### 1. Run the Self-Contained Distill + Arrow/DataFusion Lakehouse Demo Script
+
+You can run the end-to-end Python lakehouse & behavioral analytics pipeline immediately using `uv` inside the repository's Nix development shell:
+
+```bash
+nix develop -c uv run --no-project \
+ --with networkx --with pandas --with plotly --with pydantic \
+ --with dateparser --with pksuid --with pyarrow --with datafusion \
+ python examples/apache-lakehouse/distill_lakehouse_demo.py
+```
+
+This script:
+1. Normalizes a realistic **UserALE 2.5 OTLP/JSON** batch (containing `click`, `rage_click`, `mutation`, `scroll`, and `session_replay` events correlated with W3C `traceId`/`spanId` and `targetRect` geometry) via `distill.schemas.userale.extract_logs`.
+2. Writes the normalized batch to an **Apache Arrow / Apache Parquet** lakehouse table (and demonstrates registering an **Apache Iceberg** table via `pyiceberg`).
+3. Queries the table using **Apache DataFusion SQL** (`OVER (PARTITION BY session_id ORDER BY client_time)` window functions) to compute next-target transitions and per-page friction metrics.
+4. Runs **Apache Flagon Distill 2.5** analytics (`session_summary`, `frustration_score`, `detect_rage_clicks`, `detect_dead_clicks`, `detect_dom_thrash`, `transition_matrix`, and `correlate_by_trace`).
+
+---
+
+### 2. Start the Streaming Docker Compose Stack (OTel Collector + Apache Kafka + Apache Iceberg REST Catalog)
+
+```bash
+docker compose -f examples/apache-lakehouse/docker-compose.yml up -d
+```
+
+- **OTLP HTTP Endpoint**: `http://localhost:4318/v1/logs` (CORS enabled for browser `fetch` / `keepalive`)
+- **Apache Kafka Broker (KRaft)**: `localhost:9092` (topic: `flagon-userale-events`)
+- **Apache Iceberg REST Catalog**: `http://localhost:8181`
+
+To stop and clean up:
+
+```bash
+docker compose -f examples/apache-lakehouse/docker-compose.yml down -v
+```
diff --git a/examples/apache-lakehouse/distill_lakehouse_demo.py b/examples/apache-lakehouse/distill_lakehouse_demo.py
new file mode 100644
index 0000000..7bd27a5
--- /dev/null
+++ b/examples/apache-lakehouse/distill_lakehouse_demo.py
@@ -0,0 +1,208 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+End-to-End "All-Apache" Behavioral Lakehouse Demo:
+Apache Flagon UserALE.js (OTLP/JSON) -> Apache Arrow / Parquet / Iceberg ->
+Apache DataFusion Vectorized SQL -> Apache Flagon Distill 2.5 Analytics.
+"""
+
+from pathlib import Path
+import sys
+import tempfile
+
+# Allow running directly from the repository root without installing flagon-distill globally
+REPO_ROOT = Path(__file__).resolve().parents[2]
+DISTILL_PATH = REPO_ROOT / "products" / "distill"
+if str(DISTILL_PATH) not in sys.path:
+ sys.path.insert(0, str(DISTILL_PATH))
+
+import distill # noqa: E402
+from distill.schemas.userale import extract_logs # noqa: E402
+
+
+def build_sample_otlp_payload() -> dict:
+ """
+ Constructs a realistic OpenTelemetry ExportLogsServiceRequest payload emitted by
+ Apache Flagon UserALE.js 2.5 (OtlpJsonExporter), including raw clicks, dead clicks,
+ rage clicks, DOM mutation summaries, and W3C traceId/spanId correlation.
+ """
+ return {
+ "resourceLogs": [
+ {
+ "resource": {
+ "attributes": [
+ {"key": "service.name", "value": {"stringValue": "flagon-storefront"}},
+ {"key": "service.version", "value": {"stringValue": "2.5.0"}},
+ {"key": "telemetry.sdk.name", "value": {"stringValue": "apache-flagon-userale"}},
+ {"key": "telemetry.sdk.version", "value": {"stringValue": "2.4.0"}},
+ {"key": "session.id", "value": {"stringValue": "sess_lakehouse_01"}},
+ {"key": "screen.width", "value": {"intValue": "1440"}},
+ {"key": "screen.height", "value": {"intValue": "900"}},
+ {"key": "browser.name", "value": {"stringValue": "Chrome"}},
+ {"key": "browser.version", "value": {"stringValue": "130.0.0.0"}},
+ ]
+ },
+ "scopeLogs": [
+ {
+ "scope": {"name": "org.apache.flagon.userale", "version": "2.4.0"},
+ "logRecords": [
+ # 1. User clicks product card -> followed by DOM mutation
+ {
+ "timeUnixNano": "1730000001000000000",
+ "eventName": "userale.raw.click",
+ "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
+ "spanId": "00f067aa0ba902b7",
+ "attributes": [
+ {"key": "flagon.log_type", "value": {"stringValue": "raw"}},
+ {"key": "flagon.event_type", "value": {"stringValue": "click"}},
+ {"key": "flagon.target", "value": {"stringValue": "button#add-to-cart"}},
+ {"key": "url.full", "value": {"stringValue": "https://shop.example.org/product/42"}},
+ {"key": "flagon.location.x", "value": {"intValue": "720"}},
+ {"key": "flagon.location.y", "value": {"intValue": "450"}},
+ {
+ "key": "flagon.target_rect",
+ "value": {
+ "kvlistValue": {
+ "values": [
+ {"key": "x", "value": {"intValue": "680"}},
+ {"key": "y", "value": {"intValue": "430"}},
+ {"key": "width", "value": {"intValue": "80"}},
+ {"key": "height", "value": {"intValue": "40"}},
+ ]
+ }
+ },
+ },
+ ],
+ "body": {"kvlistValue": {"values": [{"key": "clicks", "value": {"intValue": "1"}}]}},
+ },
+ # 2. Throttled DOM mutation summary on cart drawer
+ {
+ "timeUnixNano": "1730000001120000000",
+ "eventName": "userale.mutation.mutation",
+ "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
+ "spanId": "00f067aa0ba902b7",
+ "attributes": [
+ {"key": "flagon.log_type", "value": {"stringValue": "mutation"}},
+ {"key": "flagon.event_type", "value": {"stringValue": "mutation"}},
+ {"key": "flagon.target", "value": {"stringValue": "aside#cart-drawer"}},
+ {"key": "url.full", "value": {"stringValue": "https://shop.example.org/product/42"}},
+ ],
+ "body": {
+ "kvlistValue": {
+ "values": [
+ {"key": "addedCount", "value": {"intValue": "14"}},
+ {"key": "removedCount", "value": {"intValue": "6"}},
+ {"key": "attributeCount", "value": {"intValue": "4"}},
+ ]
+ }
+ },
+ },
+ # 3. Three rapid clicks on an unresponsive promo banner (rage click + dead clicks)
+ *[
+ {
+ "timeUnixNano": str(1730000003000000000 + offset_ms * 1_000_000),
+ "eventName": "userale.raw.click",
+ "traceId": "8a3c60f7d188f8fa79d48a391a778fa6",
+ "spanId": f"11f067aa0ba902b{idx}",
+ "attributes": [
+ {"key": "flagon.log_type", "value": {"stringValue": "raw"}},
+ {"key": "flagon.event_type", "value": {"stringValue": "click"}},
+ {"key": "flagon.target", "value": {"stringValue": "div.promo-banner"}},
+ {"key": "url.full", "value": {"stringValue": "https://shop.example.org/checkout"}},
+ {"key": "flagon.location.x", "value": {"intValue": str(300 + idx)}},
+ {"key": "flagon.location.y", "value": {"intValue": str(180 + idx)}},
+ ],
+ "body": {},
+ }
+ for idx, offset_ms in enumerate((0, 140, 290))
+ ],
+ ],
+ }
+ ],
+ }
+ ]
+ }
+
+
+def main() -> None:
+ # 1. Normalize OTLP/JSON via Apache Flagon Distill
+ otlp_payload = build_sample_otlp_payload()
+ logs = extract_logs(otlp_payload)
+ print(f"[1/3] Ingested & normalized {len(logs)} UserALE OTLP log records via Distill.")
+
+ # 2. Write to Apache Parquet & query with Apache DataFusion (if installed)
+ try:
+ import datafusion
+ import pyarrow as pa
+ import pyarrow.parquet as pq
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ parquet_path = Path(tmpdir) / "userale_events.parquet"
+ arrow_rows = [
+ {
+ "session_id": r.get("sessionId"),
+ "client_time": int(r.get("clientTime", 0)),
+ "log_type": r.get("logType"),
+ "event_type": r.get("type"),
+ "target": r.get("target"),
+ "page_url": r.get("pageUrl"),
+ "trace_id": r.get("traceId"),
+ }
+ for r in logs
+ ]
+ table = pa.Table.from_pylist(arrow_rows)
+ pq.write_table(table, parquet_path)
+
+ ctx = datafusion.SessionContext()
+ ctx.register_parquet("userale_events", str(parquet_path))
+ df = ctx.sql(
+ """
+ SELECT
+ session_id,
+ client_time,
+ event_type,
+ target,
+ LEAD(target) OVER (PARTITION BY session_id ORDER BY client_time) AS next_target,
+ trace_id
+ FROM userale_events
+ ORDER BY client_time
+ """
+ )
+ print("\n[2/3] Apache DataFusion Vectorized SQL Window Query over Apache Parquet:")
+ df.show()
+ except ImportError:
+ print("\n[2/3] (Optional pyarrow/datafusion not installed; skipping SQL preview)")
+
+ # 3. Run Apache Flagon Distill 2.5 Behavioral & Frustration Analytics
+ summary = distill.session_summary(logs)
+ frustration = distill.frustration_score(logs)
+ transitions = distill.transition_matrix(logs, drop_recursions=True, as_dataframe=False)
+ traces = distill.correlate_by_trace(logs)
+
+ print("\n[3/3] Apache Flagon Distill 2.5 Behavioral Analytics:")
+ print(f" - Session Summary: {summary}")
+ print(
+ f" - Frustration Score: {frustration['frustrationScore']} ({frustration['severity']}) "
+ f"[rage={frustration['rageClickCount']}, dead={frustration['deadClickCount']}, "
+ f"thrash={frustration['domThrashBurstCount']}]"
+ )
+ print(f" - Markov Transitions: {transitions}")
+ print(f" - Correlated Traces: {list(traces.keys())}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/apache-lakehouse/docker-compose.yml b/examples/apache-lakehouse/docker-compose.yml
new file mode 100644
index 0000000..5082e15
--- /dev/null
+++ b/examples/apache-lakehouse/docker-compose.yml
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ # Official Apache Kafka GraalVM native image in KRaft mode (zero ZooKeeper, <1s startup, Apache-2.0)
+ kafka:
+ image: apache/kafka-native:3.8.0
+ container_name: flagon-apache-kafka
+ ports:
+ - "9092:9092"
+ environment:
+ KAFKA_NODE_ID: 1
+ KAFKA_PROCESS_ROLES: broker,controller
+ KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
+ KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+ KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+ KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
+ KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
+ KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+ KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+ KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+ KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
+
+ # Official Apache Iceberg REST Catalog backed by a local warehouse volume (Apache-2.0)
+ iceberg-rest:
+ image: apache/iceberg-rest-fixture:1.7.0
+ container_name: flagon-apache-iceberg-rest
+ ports:
+ - "8181:8181"
+ environment:
+ CATALOG_WAREHOUSE: file:///var/lib/iceberg/warehouse
+ CATALOG_IO__IMPL: org.apache.iceberg.hadoop.HadoopFileIO
+ volumes:
+ - iceberg-warehouse:/var/lib/iceberg/warehouse
+
+ # OpenTelemetry Collector receiving UserALE OTLP/JSON and streaming to Apache Kafka + local lakehouse volume
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.114.0
+ container_name: flagon-lakehouse-otel-collector
+ command: ["--config=/etc/otel-collector-config.yaml"]
+ volumes:
+ - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
+ - lakehouse-staging:/var/lib/flagon-lakehouse
+ ports:
+ - "4317:4317" # OTLP gRPC
+ - "4318:4318" # OTLP HTTP (/v1/logs, /v1/traces)
+ depends_on:
+ - kafka
+ - iceberg-rest
+
+volumes:
+ iceberg-warehouse:
+ lakehouse-staging:
diff --git a/examples/apache-lakehouse/otel-collector-config.yaml b/examples/apache-lakehouse/otel-collector-config.yaml
new file mode 100644
index 0000000..90f17f3
--- /dev/null
+++ b/examples/apache-lakehouse/otel-collector-config.yaml
@@ -0,0 +1,60 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+receivers:
+ otlp:
+ protocols:
+ grpc:
+ endpoint: 0.0.0.0:4317
+ http:
+ endpoint: 0.0.0.0:4318
+ cors:
+ allowed_origins:
+ - "*"
+ allowed_headers:
+ - "Content-Type"
+ - "Authorization"
+ - "x-api-key"
+ - "traceparent"
+ - "tracestate"
+ max_age: 7200
+
+processors:
+ batch:
+ timeout: 1s
+ send_batch_size: 512
+
+exporters:
+ kafka/flagon:
+ brokers:
+ - kafka:9092
+ protocol_version: 2.0.0
+ topic: flagon-userale-events
+ encoding: otlp_json
+
+ file/lakehouse:
+ path: /var/lib/flagon-lakehouse/userale-otlp.jsonl
+
+ debug:
+ verbosity: normal
+
+service:
+ pipelines:
+ logs:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [kafka/flagon, file/lakehouse, debug]
diff --git a/examples/opensearch/README.md b/examples/opensearch/README.md
new file mode 100644
index 0000000..d469737
--- /dev/null
+++ b/examples/opensearch/README.md
@@ -0,0 +1,100 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Apache Flagon UserALE + OpenSearch (`Apache-2.0`) Example
+
+This directory provides a 100% `Apache-2.0`-licensed **OpenTelemetry Collector + OpenSearch + OpenSearch Dashboards** deployment (for both **Docker Compose** and **Kubernetes**) as an open-source replacement for the legacy ELK (`examples/elk/`) stack.
+
+> **Looking for the All-Apache Lakehouse (`Apache Kafka` + `Apache Iceberg` + `Apache DataFusion` + `Apache Superset` + `Distill`)?** See [`../apache-lakehouse/README.md`](../apache-lakehouse/README.md).
+> **Looking for OpenTelemetry + Grafana/Loki/Tempo?** See [`../../products/userale/example/otel-demo/README.md`](../../products/userale/example/otel-demo/README.md).
+
+---
+
+## Architecture
+
+```
+Browser (UserALE.js OtlpJsonExporter)
+ │ POST http://localhost:4318/v1/logs (OTLP/JSON + W3C traceId/spanId)
+ ▼
+OpenTelemetry Collector (otelcol-contrib, Apache-2.0)
+ ├──► OpenSearch (:9200, Apache-2.0)
+ │ ├── flagon-userale-logs (Raw, Interval, Custom, Mutation, Replay logs)
+ │ └── flagon-userale-traces (Correlated W3C distributed traces)
+ ▼
+OpenSearch Dashboards (:5601, Apache-2.0)
+```
+
+---
+
+## 1. Docker Compose Quickstart
+
+### Step 1: Start OpenSearch, OpenSearch Dashboards, and the OpenTelemetry Collector
+
+```bash
+docker compose -f examples/opensearch/docker-compose.yml up -d
+```
+
+### Step 2: Apply the UserALE Index Template (Optional, Recommended for Flat JSON Ingest)
+
+```bash
+curl -X PUT "http://localhost:9200/_index_template/flagon-userale" \
+ -H "Content-Type: application/json" \
+ -d @examples/opensearch/userale-index-template.json
+```
+
+### Step 3: Send Telemetry from UserALE.js
+
+Configure UserALE.js in your application (or open `products/userale/example/otel-demo/index.html`) to export OTLP/JSON to `http://localhost:4318/v1/logs`:
+
+```javascript
+import * as userale from "flagon-userale";
+
+userale.options({
+ url: "http://localhost:4318/v1/logs",
+ exporterFormat: "otlp-json",
+ toolName: "my-web-app",
+ toolVersion: "1.0.0",
+ observeMutations: true,
+ detectRageClicks: true,
+});
+```
+
+### Step 4: Explore in OpenSearch Dashboards
+
+1. Open [http://localhost:5601](http://localhost:5601).
+2. Go to **Stack Management -> Index Patterns** and create an index pattern matching `flagon-userale*` using `@timestamp` (or `clientTime` for direct JSON ingestion) as the time field.
+3. Open **Discover** to query clicks, rage-clicks (`attributes.flagon.event_type: "rage_click"`), DOM mutations (`attributes.flagon.log_type: "mutation"`), and correlated `traceId` spans.
+
+### Step 5: Tear Down
+
+```bash
+docker compose -f examples/opensearch/docker-compose.yml down -v
+```
+
+---
+
+## 2. Kubernetes Deployment
+
+To deploy the OpenTelemetry Collector + OpenSearch + OpenSearch Dashboards stack in a Kubernetes cluster (such as `kind`, `minikube`, or an enterprise cluster):
+
+```bash
+kubectl apply -f examples/opensearch/kubernetes/opensearch.yaml
+kubectl -n flagon-opensearch port-forward svc/otel-collector 4318:4318 &
+kubectl -n flagon-opensearch port-forward svc/opensearch-dashboards 5601:5601 &
+```
diff --git a/examples/opensearch/docker-compose.yml b/examples/opensearch/docker-compose.yml
new file mode 100644
index 0000000..e618359
--- /dev/null
+++ b/examples/opensearch/docker-compose.yml
@@ -0,0 +1,64 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+services:
+ opensearch:
+ image: opensearchproject/opensearch:2.18.0
+ container_name: flagon-opensearch
+ environment:
+ - discovery.type=single-node
+ - DISABLE_SECURITY_PLUGIN=true
+ - DISABLE_INSTALL_DEMO_CONFIG=true
+ - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ nofile:
+ soft: 65536
+ hard: 65536
+ volumes:
+ - opensearch-data:/usr/share/opensearch/data
+ ports:
+ - "9200:9200"
+ - "9600:9600"
+
+ opensearch-dashboards:
+ image: opensearchproject/opensearch-dashboards:2.18.0
+ container_name: flagon-opensearch-dashboards
+ environment:
+ - 'OPENSEARCH_HOSTS=["http://opensearch:9200"]'
+ - DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
+ ports:
+ - "5601:5601"
+ depends_on:
+ - opensearch
+
+ otel-collector:
+ image: otel/opentelemetry-collector-contrib:0.114.0
+ container_name: flagon-opensearch-otel-collector
+ command: ["--config=/etc/otel-collector-config.yaml"]
+ volumes:
+ - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
+ ports:
+ - "4317:4317" # OTLP gRPC
+ - "4318:4318" # OTLP HTTP (/v1/logs, /v1/traces)
+ depends_on:
+ - opensearch
+
+volumes:
+ opensearch-data:
diff --git a/examples/opensearch/kubernetes/opensearch.yaml b/examples/opensearch/kubernetes/opensearch.yaml
new file mode 100644
index 0000000..60d337d
--- /dev/null
+++ b/examples/opensearch/kubernetes/opensearch.yaml
@@ -0,0 +1,188 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: flagon-opensearch
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: otel-collector-config
+ namespace: flagon-opensearch
+data:
+ otel-collector-config.yaml: |
+ receivers:
+ otlp:
+ protocols:
+ grpc:
+ endpoint: 0.0.0.0:4317
+ http:
+ endpoint: 0.0.0.0:4318
+ cors:
+ allowed_origins: ["*"]
+ allowed_headers: ["Content-Type", "Authorization", "x-api-key", "traceparent", "tracestate"]
+ processors:
+ batch:
+ timeout: 1s
+ send_batch_size: 512
+ exporters:
+ opensearch/logs:
+ http:
+ endpoint: http://opensearch:9200
+ tls:
+ insecure: true
+ logs_index: flagon-userale-logs
+ service:
+ pipelines:
+ logs:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [opensearch/logs]
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+ name: opensearch
+ namespace: flagon-opensearch
+spec:
+ serviceName: opensearch
+ replicas: 1
+ selector:
+ matchLabels:
+ app: opensearch
+ template:
+ metadata:
+ labels:
+ app: opensearch
+ spec:
+ containers:
+ - name: opensearch
+ image: opensearchproject/opensearch:2.18.0
+ env:
+ - name: discovery.type
+ value: single-node
+ - name: DISABLE_SECURITY_PLUGIN
+ value: "true"
+ - name: OPENSEARCH_JAVA_OPTS
+ value: "-Xms512m -Xmx512m"
+ ports:
+ - containerPort: 9200
+ name: http
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: opensearch
+ namespace: flagon-opensearch
+spec:
+ selector:
+ app: opensearch
+ ports:
+ - port: 9200
+ targetPort: 9200
+ name: http
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: opensearch-dashboards
+ namespace: flagon-opensearch
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: opensearch-dashboards
+ template:
+ metadata:
+ labels:
+ app: opensearch-dashboards
+ spec:
+ containers:
+ - name: dashboards
+ image: opensearchproject/opensearch-dashboards:2.18.0
+ env:
+ - name: OPENSEARCH_HOSTS
+ value: '["http://opensearch:9200"]'
+ - name: DISABLE_SECURITY_DASHBOARDS_PLUGIN
+ value: "true"
+ ports:
+ - containerPort: 5601
+ name: http
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: opensearch-dashboards
+ namespace: flagon-opensearch
+spec:
+ selector:
+ app: opensearch-dashboards
+ ports:
+ - port: 5601
+ targetPort: 5601
+ name: http
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: otel-collector
+ namespace: flagon-opensearch
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: otel-collector
+ template:
+ metadata:
+ labels:
+ app: otel-collector
+ spec:
+ containers:
+ - name: otel-collector
+ image: otel/opentelemetry-collector-contrib:0.114.0
+ args: ["--config=/etc/otel-collector-config.yaml"]
+ volumeMounts:
+ - name: config
+ mountPath: /etc/otel-collector-config.yaml
+ subPath: otel-collector-config.yaml
+ ports:
+ - containerPort: 4317
+ name: otlp-grpc
+ - containerPort: 4318
+ name: otlp-http
+ volumes:
+ - name: config
+ configMap:
+ name: otel-collector-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: otel-collector
+ namespace: flagon-opensearch
+spec:
+ selector:
+ app: otel-collector
+ ports:
+ - port: 4317
+ targetPort: 4317
+ name: otlp-grpc
+ - port: 4318
+ targetPort: 4318
+ name: otlp-http
diff --git a/examples/opensearch/otel-collector-config.yaml b/examples/opensearch/otel-collector-config.yaml
new file mode 100644
index 0000000..9767939
--- /dev/null
+++ b/examples/opensearch/otel-collector-config.yaml
@@ -0,0 +1,68 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+receivers:
+ otlp:
+ protocols:
+ grpc:
+ endpoint: 0.0.0.0:4317
+ http:
+ endpoint: 0.0.0.0:4318
+ cors:
+ allowed_origins:
+ - "*"
+ allowed_headers:
+ - "Content-Type"
+ - "Authorization"
+ - "x-api-key"
+ - "traceparent"
+ - "tracestate"
+ max_age: 7200
+
+processors:
+ batch:
+ timeout: 1s
+ send_batch_size: 512
+
+exporters:
+ opensearch/logs:
+ http:
+ endpoint: http://opensearch:9200
+ tls:
+ insecure: true
+ logs_index: flagon-userale-logs
+
+ opensearch/traces:
+ http:
+ endpoint: http://opensearch:9200
+ tls:
+ insecure: true
+ traces_index: flagon-userale-traces
+
+ debug:
+ verbosity: normal
+
+service:
+ pipelines:
+ logs:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [opensearch/logs, debug]
+ traces:
+ receivers: [otlp]
+ processors: [batch]
+ exporters: [opensearch/traces, debug]
diff --git a/examples/opensearch/userale-index-template.json b/examples/opensearch/userale-index-template.json
new file mode 100644
index 0000000..4a32ff3
--- /dev/null
+++ b/examples/opensearch/userale-index-template.json
@@ -0,0 +1,82 @@
+{
+ "index_patterns": ["flagon-userale*"],
+ "template": {
+ "settings": {
+ "number_of_shards": 1,
+ "number_of_replicas": 0,
+ "index.mapping.total_fields.limit": 2000
+ },
+ "mappings": {
+ "dynamic_templates": [
+ {
+ "details_strings_as_keywords": {
+ "path_match": "details.*",
+ "match_mapping_type": "string",
+ "mapping": {
+ "type": "keyword",
+ "ignore_above": 1024
+ }
+ }
+ }
+ ],
+ "properties": {
+ "@timestamp": { "type": "date" },
+ "clientTime": { "type": "date", "format": "epoch_millis||strict_date_optional_time" },
+ "startTime": { "type": "date", "format": "epoch_millis||strict_date_optional_time" },
+ "endTime": { "type": "date", "format": "epoch_millis||strict_date_optional_time" },
+ "microTime": { "type": "double" },
+ "target": { "type": "keyword" },
+ "path": { "type": "keyword" },
+ "pageUrl": { "type": "keyword" },
+ "pageTitle": { "type": "text", "fields": { "keyword": { "type": "keyword" } } },
+ "pageReferrer": { "type": "keyword" },
+ "type": { "type": "keyword" },
+ "logType": { "type": "keyword" },
+ "userAction": { "type": "boolean" },
+ "userId": { "type": "keyword" },
+ "sessionId": { "type": "keyword" },
+ "httpSessionId": { "type": "keyword" },
+ "browserSessionId": { "type": "keyword" },
+ "traceId": { "type": "keyword" },
+ "spanId": { "type": "keyword" },
+ "toolName": { "type": "keyword" },
+ "toolVersion": { "type": "keyword" },
+ "useraleVersion": { "type": "keyword" },
+ "count": { "type": "integer" },
+ "duration": { "type": "long" },
+ "targetChange": { "type": "boolean" },
+ "typeChange": { "type": "boolean" },
+ "location": {
+ "properties": {
+ "x": { "type": "float" },
+ "y": { "type": "float" }
+ }
+ },
+ "targetRect": {
+ "properties": {
+ "x": { "type": "float" },
+ "y": { "type": "float" },
+ "width": { "type": "float" },
+ "height": { "type": "float" }
+ }
+ },
+ "scrnRes": {
+ "properties": {
+ "width": { "type": "integer" },
+ "height": { "type": "integer" }
+ }
+ },
+ "browser": {
+ "properties": {
+ "browser": { "type": "keyword" },
+ "version": { "type": "keyword" }
+ }
+ },
+ "frames": {
+ "type": "object",
+ "enabled": false
+ }
+ }
+ }
+ }
+}