You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flagon.apache.org by au...@apache.org on 2026/10/11 18:10:59 UTC
(flagon) 01/01: docs: update monorepo READMEs, CONTRIBUTING.md, and site docs (#36, #47, #99)
This is an automated email from the ASF dual-hosted git repository.
brucearctor pushed a commit to branch docs/update-readmes-and-site-docs
in repository https://gitbox.apache.org/repos/asf/flagon.git
commit 3c1678974b3396fa15cb4d798ae40725f76526ef
Author: Bruce Arctor <50...@users.noreply.github.com>
AuthorDate: Sun Oct 11 11:10:11 2026 -0700
docs: update monorepo READMEs, CONTRIBUTING.md, and site docs (#36, #47, #99)
---
.gitignore | 5 +-
CONTRIBUTING.md | 105 +++++++
README.md | 92 +++++-
examples/elk/docker/README.md | 56 +++-
lefthook.yml | 3 +-
products/distill/README.md | 156 +++-------
products/userale/README.md | 58 +++-
.../userale/packages/flagon-userale-ext/README.md | 51 ++--
site/_docs/contributing.md | 77 +++--
site/_docs/distill/index.md | 60 +++-
site/_docs/useralejs/API.md | 221 +++++++++-----
site/_docs/useralejs/build.md | 153 ++++------
site/_docs/useralejs/dataschema.md | 317 ++++++---------------
site/_docs/useralejs/index.md | 156 +++++-----
site/_docs/useralejs/modifying.md | 151 ++--------
site/_docs/useralejs/troubleshooting.md | 38 ++-
16 files changed, 908 insertions(+), 791 deletions(-)
diff --git a/.gitignore b/.gitignore
index b70cff6..e31e4a9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,4 +33,7 @@ pnpm-lock.yaml
# Nix / direnv
result
-.direnv/
\ No newline at end of file
+.direnv/
+
+# Graphify local knowledge graph
+graphify-out/
\ No newline at end of file
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..64f2b8a
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,105 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+# Contributing to Apache Flagon
+
+Thank you for your interest in contributing to **Apache Flagon**! All Flagon products (**UserALE.js**, **UserALE Browser Extension**, **Distill**, **Protobuf schemas**, and the **Project Website**) live in this unified monorepo (`https://github.com/apache/flagon`).
+
+---
+
+## 1. Open or Claim a GitHub Issue
+
+All non-trivial changes should reference a [GitHub Issue](https://github.com/apache/flagon/issues). If an issue does not yet exist for your bug report or feature proposal, open one first so the community can discuss the design.
+
+---
+
+## 2. Set Up Your Development Environment
+
+### Option A: Nix Flake (Recommended)
+If you have [Nix](https://nixos.org/) with flakes enabled, run:
+
+```bash
+nix develop
+```
+
+This drops you into a reproducible shell containing Node.js, `pnpm`, TypeScript, Python 3, `buf`, `nixpkgs-fmt`, and `lefthook` git hooks.
+
+### Option B: Manual Toolchain Setup
+- **UserALE (`products/userale`)**: Node.js `>= 20.x` and `pnpm >= 10.x`
+- **Distill (`products/distill`)**: Python `>= 3.8` (`3.10+` recommended) and `poetry` (or `uv`)
+- **Protobuf (`proto/`)**: [`buf` CLI](https://buf.build/docs/installation) `>= 1.50`
+- **Website (`site/`)**: Ruby + Bundler + Jekyll
+
+---
+
+## 3. Working on Components
+
+### UserALE.js & Browser Extension (`products/userale`)
+```bash
+cd products/userale
+pnpm install
+
+# Build both flagon-userale and flagon-userale-ext
+pnpm build
+
+# Run Jest unit & contract tests
+pnpm --dir packages/flagon-userale test
+
+# Run Playwright end-to-end tests
+pnpm test
+
+# Format and lint
+pnpm format
+pnpm lint
+```
+
+### Protobuf Schemas (`proto/`)
+The canonical UserALE v1 Protobuf schema is defined in [`proto/apache/flagon/userale/v1/log.proto`](./proto/apache/flagon/userale/v1/log.proto). If you modify `.proto` files:
+
+```bash
+buf lint
+buf generate
+```
+
+Ensure `buf generate --exit-code` passes cleanly (do not reformat generated files under `products/userale/packages/flagon-userale/src/gen/`).
+
+### Distill (`products/distill`)
+```bash
+cd products/distill
+poetry install --all-groups
+poetry run pytest
+```
+
+---
+
+## 4. Pre-Commit Hooks & Code Style
+
+This repository uses [`lefthook`](./lefthook.yml) to run formatting and lint checks before commits and pushes:
+- **Prettier & ESLint** run automatically on staged files under `products/userale/`.
+- **Apache License Headers**: Every new source file must include the standard ASF Apache-2.0 license header (enforced in CI via SkyWalking Eyes).
+- Always run pre-commit hooks when committing (do not use `--no-verify`).
+
+---
+
+## 5. Submitting a Pull Request
+
+1. Create a descriptive topic branch from `master` (e.g. `feat/otel-exporter`, `fix/autostart-init`, `docs/update-readme`).
+2. Make focused commits following Conventional Commits style (e.g. `feat(userale): ...`, `fix(distill): ...`, `docs: ...`).
+3. Open a Pull Request against `apache/flagon` `master` referencing the relevant GitHub issue(s) (e.g. `Closes #123`).
+4. Ensure all CI checks (`License check`, `proto-check`, `Userale CI test suite`, and `Distill` tests) pass.
diff --git a/README.md b/README.md
index 26f1766..ba8bbd2 100644
--- a/README.md
+++ b/README.md
@@ -19,26 +19,88 @@
# Apache Flagon
-Flagon is a collection of open-source tools for user behavior analytics, including instrumentation libraries and analytics pipelines. This monorepo contains multiple Flagon products in a unified codebase.
+[Apache Flagon](https://flagon.apache.org/) is an open-source software suite for user behavioral analytics, combining lightweight frontend DOM/event instrumentation (**UserALE.js**) with OpenTelemetry observability exports, canonical Protobuf schemas, and a Python behavioral analytics & graph toolkit (**Distill**).
+
+---
## Repository Structure
+
```text
-/examples # Integration examples
-/products
-├── userale
-│ └── packages
-│ ├── flagon-userale # JavaScript client instrumentation library
-│ └── flagon-userale-ext # Browser extension for deploying UserALE
-└── distill # Data processing toolkit (Python)
-/site # Code for https://flagon.apache.org/
+├── proto/ # Canonical Protobuf schemas (apache.flagon.userale.v1)
+├── products/
+│ ├── userale/
+│ │ ├── packages/
+│ │ │ ├── flagon-userale/ # UserALE.js core TypeScript/JS client library
+│ │ │ └── flagon-userale-ext/ # Chrome/Firefox MV3 browser extension (Plasmo)
+│ │ └── example/
+│ │ ├── server/ # Local HTTP/WebSocket/OTLP collector example server
+│ │ └── otel-demo/ # Full-stack OTel Collector + Grafana + Loki + Tempo demo
+│ └── distill/ # Python behavioral analytics & graph toolkit
+├── examples/
+│ └── elk/ # Docker Compose & Kubernetes ELK stack examples
+└── site/ # Jekyll source for https://flagon.apache.org/
```
-## Products
+---
+
+## Products & Capabilities
+
+### 1. [Apache Flagon UserALE.js](./products/userale/packages/flagon-userale/README.md)
+Zero-dependency client-side instrumentation library (`ESM` + `IIFE` script tag) that automatically captures DOM interactions, interval summaries, and custom application events.
+
+- **Multiple Wire Exporters**:
+ - **UserALE Flat JSON (`'userale-json'`)**: Default backward-compatible JSON array batch transport over `fetch` (with 48 KiB chunked `keepalive: true` delivery on `pagehide`).
+ - **OpenTelemetry OTLP/JSON (`'otlp-json'`)**: Native `ExportLogsServiceRequest` exporter (`OtlpJsonExporter`) with `Resource` attribute hoisting, nanosecond `BigInt` timestamps, OTel Event naming (`eventName` / `event.name`), structured `kvlistValue` body, and W3C TraceContext (`traceId`, `spanId`) correlation.
+ - **OpenTelemetry JS SDK Bridge (`OtelSdkExporter`)**: Routes UserALE logs directly into an existing `@opentelemetry/api-logs` `Logger`.
+ - **Protobuf v1 Canonical JSON (`'proto-json'`)**: Serializes batches conforming to [`proto/apache/flagon/userale/v1/log.proto`](./proto/apache/flagon/userale/v1/log.proto) (`apache.flagon.userale.v1.LogBatch`).
+ - **WebSocket (`'ws'`)**: Persistent streaming transport for `ws://` and `wss://` endpoints.
+
+### 2. [UserALE Browser Extension](./products/userale/packages/flagon-userale-ext/README.md)
+Manifest V3 browser extension for Chrome and Firefox that injects UserALE into pages matching a configurable URL allowlist, complete with options for **Exporter Format** (`UserALE JSON`, `OpenTelemetry (OTLP/JSON)`, `Protobuf v1 JSON`), HTTP Basic Auth, OAuth PKCE, and custom JSON headers.
+
+### 3. [Apache Flagon Distill](./products/distill/README.md)
+Python behavioral analytics package (`flagon-distill`) for session segmentation, funnel conversion analysis, and directed clickstream graph modeling. Seamlessly ingests UserALE flat JSON (`v2.3+`), Protobuf v1 `LogBatch` JSON, and OpenTelemetry OTLP/JSON (`ExportLogsServiceRequest`) payloads.
+
+---
+
+## Quick Start
+
+### Development Environment (Nix)
+This repository includes a reproducible [`flake.nix`](./flake.nix) providing Node.js, `pnpm`, TypeScript, Python, `buf`, and `lefthook`:
+
+```bash
+nix develop
+```
+
+### Build & Test UserALE.js
+```bash
+cd products/userale
+pnpm install
+pnpm build
+pnpm --dir packages/flagon-userale test
+```
+
+### Run the 5-Minute OpenTelemetry + Grafana Demo
+Spin up an OpenTelemetry Collector, Loki, Tempo, and pre-provisioned Grafana dashboard correlating browser clicks with distributed traces:
+
+```bash
+cd products/userale/example/otel-demo
+docker compose up -d
+# Open http://localhost:8080 (demo app) and http://localhost:3000 (Grafana)
+```
+
+### Build & Test Distill (Python)
+```bash
+cd products/distill
+poetry install --all-groups
+poetry run pytest
+```
-- [UserALE Core Library](./products/userale/packages/flagon-userale/README.md)
-- [UserALE Browser Extension](./products/userale/packages/flagon-userale-ext/README.md)
-- [Distill Toolkit](./products/distill/README.md)
+---
-## Documentation
+## Documentation & Community
-- 📚 [Flagon Website](https://flagon.apache.org/)
\ No newline at end of file
+- 📚 [Apache Flagon Website & Docs](https://flagon.apache.org/)
+- 🤝 [Contributing Guide](./CONTRIBUTING.md)
+- 🐛 [GitHub Issues](https://github.com/apache/flagon/issues)
+- 💬 [Mailing Lists](https://flagon.apache.org/community/) (`dev@flagon.apache.org`)
\ No newline at end of file
diff --git a/examples/elk/docker/README.md b/examples/elk/docker/README.md
index 757057a..5911171 100644
--- a/examples/elk/docker/README.md
+++ b/examples/elk/docker/README.md
@@ -17,12 +17,58 @@
~ under the License.
-->
-# Example Docker deployment
+# Apache Flagon UserALE + ELK Stack Docker Example
-This directory contains a modified [elk stack](https://github.com/deviantony/docker-elk) along with an instrumented site
-to demonstrate how to publish logs to elk.
+This directory contains a pre-configured **Elasticsearch + Logstash + Kibana (ELK)** Docker Compose deployment adapted from [`deviantony/docker-elk`](https://github.com/deviantony/docker-elk) alongside a containerized example web page instrumented with **Apache Flagon UserALE.js**.
-Prerequisites: Docker engine is running
+> **Looking for OpenTelemetry + Grafana/Loki/Tempo?** See [`products/userale/example/otel-demo/`](../../../products/userale/example/otel-demo/README.md) for the modern OpenTelemetry Collector demo stack.
-Run `docker-compose up setup` to initialize and `docker-compose up` to run.
+---
+## How This Differs from Stock `docker-elk`
+
+1. **Logstash HTTP Input & UserALE Pipeline ([`logstash/pipeline/logstash.conf`](./logstash/pipeline/logstash.conf))**:
+ - Exposes an HTTP input listener on port `8000` with CORS headers enabled so browser pages instrumented with UserALE.js can `POST` JSON log batches directly.
+ - Splits batched JSON arrays into individual event documents and strips internal Logstash transport metadata before indexing into Elasticsearch.
+2. **Elasticsearch Index Template ([`logstash/templates/userale.json`](./logstash/templates/userale.json))**:
+ - Maps UserALE fields (`target`, `path`, `pageUrl`, `pageTitle`, `clientTime`, `microTime`, `location`, `scrnRes`, `type`, `logType`, `userAction`, `details`, `userId`, `sessionId`, `toolName`, `toolVersion`, `useraleVersion`) with appropriate keyword, numeric, and date types in the `flagon-userale` index.
+3. **Instrumented Example Site ([`site/Dockerfile`](./site/Dockerfile))**:
+ - Builds and serves the UserALE example web application on `http://localhost:8080`, pre-configured to send behavioral telemetry to Logstash at `http://localhost:8000`.
+
+---
+
+## Prerequisites
+
+- Docker Engine & Docker Compose (`docker compose` or `docker-compose`)
+- At least 4 GB of RAM allocated to Docker for Elasticsearch
+
+---
+
+## Running the Example
+
+1. **Initialize Elasticsearch users and roles**:
+
+```bash
+docker compose up setup
+```
+
+2. **Start the full stack (Elasticsearch, Logstash, Kibana, and the instrumented example site)**:
+
+```bash
+docker compose up -d
+```
+
+3. **Generate Behavioral Telemetry**:
+ - Open [`http://localhost:8080`](http://localhost:8080) in your browser.
+ - Click buttons, type into form inputs, hover, and scroll to emit `raw`, `interval`, and `custom` UserALE logs to Logstash (`http://localhost:8000`).
+
+4. **Explore & Query Logs in Kibana**:
+ - Open [`http://localhost:5601`](http://localhost:5601) and log in with the default credentials defined in [`.env`](./.env) (`elastic` / `changeme`).
+ - Navigate to **Stack Management -> Index Patterns** (or **Data Views**) and create a pattern matching `flagon-userale*` using `clientTime` as the timestamp field.
+ - Open **Discover** to inspect and filter live UserALE interaction events.
+
+5. **Stop the Stack**:
+
+```bash
+docker compose down -v
+```
diff --git a/lefthook.yml b/lefthook.yml
index c45d713..2881ca3 100644
--- a/lefthook.yml
+++ b/lefthook.yml
@@ -35,7 +35,8 @@ pre-push:
commands:
userale-build:
root: "products/userale/packages/flagon-userale/"
- run: npx tsup --onSuccess 'tsc --emitDeclarationOnly --declaration'
+ run: pnpm build
userale-test:
root: "products/userale/packages/flagon-userale/"
run: npx jest -c ./test/jest.config.ts
+
diff --git a/products/distill/README.md b/products/distill/README.md
index bfbb199..43a7175 100644
--- a/products/distill/README.md
+++ b/products/distill/README.md
@@ -19,145 +19,63 @@
# Apache Flagon Distill
-[](https://incubator-flagon-distill.readthedocs.io/en/distill_toolkit_refactor/?badge=stable)
+**Apache Flagon Distill** (`flagon-distill`) is a Python behavioral analytics framework for transforming, segmenting, and modeling frontend user interaction logs generated by [Apache Flagon UserALE.js](../userale/packages/flagon-userale/README.md).
-This project is a work in progress, prior to an official Apache Software
-Foundation release. Check back soon for important updates.
+- 📚 [Apache Flagon Website](https://flagon.apache.org/)
+- 🤝 [Contributing Guide](../../CONTRIBUTING.md)
-Please see our [readthedocs.org
-pages](https://incubator-flagon-distill.readthedocs.io/en/distill_toolkit_refactor/)
-for documentation.
+---
-A contribution guide has been provided
-[here](http://flagon.incubator.apache.org/docs/contributing/).
+## Key Capabilities
-## Installation
+- **Multi-Format Log Ingestion (`distill.schemas.userale`)**:
+ - **UserALE Flat JSON (`v2.3` / `v2.4` / `v2.5`)**: Validates and normalizes standard UserALE log objects (`raw`, `interval`, `custom`), including floating-point `microTime`, `httpSessionId`, `browserSessionId`, and arbitrary `details` maps.
+ - **Protobuf v1 Canonical JSON (`LogBatch`)**: Automatically unwraps and normalizes `apache.flagon.userale.v1.LogBatch` / `UserALELog` payloads emitted by UserALE's `ProtoJsonExporter`.
+ - **OpenTelemetry OTLP/JSON (`ExportLogsServiceRequest`)**: Automatically extracts and reconstructs UserALE logs from `resourceLogs[].scopeLogs[].logRecords[]` emitted by UserALE's `OtlpJsonExporter`.
+- **Behavioral Segmentation (`distill.segmentation`)**: Slice and group interaction streams by session, time window, event type, or custom predicates.
+- **Process & Graph Analytics (`distill.process`, `distill.analytics.graph`)**: Build directed clickstream transition graphs and analyze user navigation paths with NetworkX, Plotly, and Matplotlib.
-To install and set up the Python project, Distill uses
-[Poetry](https://python-poetry.org/), a dependency and package
-management tool. Poetry simplifies the management of project
-dependencies and virtual environments, ensuring consistent and
-reproducible builds.
+---
-### Prerequisites
-
-Before you begin, make sure you have the following prerequisites
-installed on your system:
+## Multi-Format Ingestion Quick Start
-- Python (\>= 3.8)
-- Poetry (\>= 1.0)
+```python
+from distill.process.log import Log
+from distill.schemas.userale import UserAleSchema, extract_logs
-You can check your Python version by running:
+# 1. Single log or batch of flat UserALE JSON, Proto-JSON LogBatch, or OTLP/JSON:
+raw_docs = extract_logs(payload) # Accepts list[dict], LogBatch dict, or OTLP ExportLogsServiceRequest dict
+validated = [UserAleSchema(**doc) for doc in raw_docs]
-``` bash
-python --version
+# 2. Or load directly into a Distill Log container:
+log_stream = Log(data=payload, schema=UserAleSchema)
```
-This will return the version of Python installed on your system. If you
-do not have Python installed, you can download it from the [official
-website](https://www.python.org/downloads/). However, we recommend using
-a Python version manager such as [pyenv]{.title-ref}. You can refer to
-this guide for setting it up: [pyenv
-guide](https://realpython.com/intro-to-pyenv/).
+---
-You can install Poetry a number of ways (see the [Poetry
-docs](https://python-poetry.org/docs/) for all methods). We recommend
-installing one of the following two ways:
+## Installation & Development
-**Official Installer**:
-
-*Linux, macOS, Windows (WSL)*
+### Prerequisites
+- Python `>= 3.8,< 4.0`
+- [Poetry](https://python-poetry.org/) `>= 1.0` (or [`uv`](https://docs.astral.sh/uv/) via `nix develop`)
-``` bash
-curl -sSL https://install.python-poetry.org | python3 -
-```
+### Setup from the Monorepo
-*Windows (Powershell)*
+1. Clone the repository and navigate to `products/distill`:
-``` bash
-(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
+```bash
+git clone https://github.com/apache/flagon.git
+cd flagon/products/distill
```
-**pipx**:
+2. Install dependencies (including test and documentation groups):
-``` bash
-pipx install poetry
+```bash
+poetry install --all-groups
```
-The above two methods should minimize the chances of dependency
-conflicts with your system Python (global) installation. Some users have
-reported issues with Poetry using an incorrect Python environment
-instead of the project\'s local virtual environment when using regular
-pip method. If you run into issues, please refer to the official Poetry
-docs or Github for more in-depth installation instructions.
-
-### Installation Steps
-
-Follow these steps to set up and install the project:
-
-1. Clone the repository:
-
- > ``` bash
- > git clone https://github.com/apache/flagon-distill.git
- > ```
-
-2. Navigate to the project directory:
-
- > ``` bash
- > cd flagon-distill
- > ```
-
-3. Use Poetry to install project dependencies and create a virtual
- environment:
-
- > ``` bash
- > poetry install
- > ```
-
- This command reads the `pyproject.toml` file and installs all
- required packages into a dedicated virtual environment.
-
-4. If you would like to run tests or make changes, install optional
- dependency groups.
+3. Run the test suite:
- > ``` bash
- > poetry install --all-groups
- > ```
-
-5. Run the tests:
-
- You can now run the tests to make sure everything installed
- properly. For example:
-
- > ``` bash
- > make test
- > ```
-
- Remember that you need to activate the virtual environment (step 4)
- each time you work on the project.
-
-### Updating Dependencies
-
-To update project dependencies, you can use the following command:
-
-``` bash
-poetry update
+```bash
+poetry run pytest
```
-
-This command updates the `pyproject.toml` file with the latest
-compatible versions of the packages.
-
-### Uninstalling
-
-To uninstall the project and its dependencies, simply deactivate the
-virtual environment (if activated) by typing:
-
-``` bash
-exit
-```
-
-This will exit the virtual environment. You can then safely delete the
-project directory.
-
-By following these installation steps, you can easily set up and manage
-the Python project using Poetry. Enjoy coding!
diff --git a/products/userale/README.md b/products/userale/README.md
index 0006759..3e5cd0d 100644
--- a/products/userale/README.md
+++ b/products/userale/README.md
@@ -1,13 +1,57 @@
-# Flagon UserALE
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
-UserALE is a client-side instrumentation library for behavioral analytics.
+ http://www.apache.org/licenses/LICENSE-2.0
-This directory contains two packages:
-- [`flagon-userale`](./packages/flagon-userale): Core JavaScript library
-- [`flagon-userale-ext`](./packages/flagon-userale-ext): Browser extension
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+ either express or implied. See the License for the specific
+ language governing permissions and limitations under the License.
+-->
-For details on each, see their respective READMEs.
+# Apache Flagon UserALE Workspace
-📚 [Documentation](https://flagon.incubator.apache.org/userale/)
+UserALE is a client-side behavioral analytics instrumentation library and browser extension for web applications.
+
+## Workspace Packages & Examples
+
+- [`packages/flagon-userale`](./packages/flagon-userale/README.md): Core TypeScript/JavaScript library (`ESM` module + `IIFE` script-tag bundle) with pluggable exporters for UserALE flat JSON, OpenTelemetry OTLP/JSON (`OtlpJsonExporter`), OpenTelemetry JS SDK (`OtelSdkExporter`), Protobuf v1 canonical JSON (`ProtoJsonExporter`), and WebSockets (`WebSocketExporter`).
+- [`packages/flagon-userale-ext`](./packages/flagon-userale-ext/README.md): Chrome and Firefox Manifest V3 browser extension built with [Plasmo](https://docs.plasmo.com/).
+- [`example/server`](./example/server/server.js): Local Express + WebSocket test collector supporting `/` and `/v1/logs` across UserALE flat JSON, Protobuf v1 `LogBatch`, and OTLP/JSON payloads.
+- [`example/otel-demo`](./example/otel-demo/README.md): Full-stack `docker-compose` stack with OpenTelemetry Collector, Grafana, Loki, and Tempo demonstrating browser-to-backend W3C `traceparent` correlation.
---
+
+## Workspace Commands
+
+From `products/userale`:
+
+```bash
+# Install dependencies across all workspace packages
+pnpm install
+
+# Build flagon-userale and flagon-userale-ext
+pnpm build
+
+# Run flagon-userale Jest unit & Protobuf/OTLP contract tests
+pnpm --dir packages/flagon-userale test
+
+# Run full Playwright E2E + Jest test suite
+pnpm test
+
+# Run local example collector server on http://localhost:8000
+pnpm example:run
+
+# Format and lint source files
+pnpm format
+pnpm lint
+```
+
+📚 [Apache Flagon Documentation](https://flagon.apache.org/docs/useralejs/)
diff --git a/products/userale/packages/flagon-userale-ext/README.md b/products/userale/packages/flagon-userale-ext/README.md
index 9729cfd..ea58e2a 100644
--- a/products/userale/packages/flagon-userale-ext/README.md
+++ b/products/userale/packages/flagon-userale-ext/README.md
@@ -18,63 +18,70 @@
# Flagon UserALE Browser Extension
-This package provides a browser extension for [Flagon UserALE](https://flagon.apache.org/userale/), enabling effortless, no-code instrumentation of websites. It captures user interactions and sends behavioral logs to a logging endpoint.
+This package provides a Manifest V3 browser extension for [Apache Flagon UserALE](https://flagon.apache.org/userale/), enabling zero-code behavioral instrumentation of websites. It captures DOM and browser tab interactions and exports logs to your configured collector endpoint.
-The extension is built with [Plasmo](https://docs.plasmo.com/) and supports modern browser environments like Chrome and Firefox.
+The extension is built with [Plasmo](https://docs.plasmo.com/) and supports Chrome, Edge, and Firefox.
---
## Features
-✅ Passive user interaction logging
-✅ No code changes required on instrumented pages
-✅ Configurable via extension options (logging endpoint, user ID, tool metadata)
-✅ Supports local development and custom deployments
+- Passive DOM and browser tab interaction logging across allowlisted URLs
+- Zero code changes required on instrumented web pages
+- Configurable **Exporter Format**:
+ - **UserALE JSON (Default)** (`userale-json`)
+ - **OpenTelemetry (OTLP/JSON)** (`otlp-json` — automatically normalizes `/v1/logs` endpoints)
+ - **Protobuf v1 JSON (`LogBatch`)** (`proto-json`)
+- Built-in support for **HTTP Basic Auth**, **OAuth 2.0 PKCE**, and **Custom HTTP Headers (JSON)**
---
## Getting Started (Development)
-1. Install dependencies:
+From `products/userale`:
+
+1. Install workspace dependencies and build `flagon-userale`:
```bash
pnpm install
+pnpm --dir packages/flagon-userale build
```
-2. Start the development build:
+2. Start the extension development build:
```bash
-pnpm dev
+pnpm --dir packages/flagon-userale-ext dev
```
3. Load the extension in your browser:
- - **Chrome**: Visit `chrome://extensions/`, enable Developer Mode, click "Load unpacked", and select the `build/chrome-mv3-dev/` directory.
- - **Firefox**: Visit `about:debugging`, "This Firefox", "Load Temporary Add-on", and select the `manifest.json` file in `build/firefox-mv3-dev/`.
+ - **Chrome / Edge**: Visit `chrome://extensions/`, enable **Developer mode**, click **Load unpacked**, and select `packages/flagon-userale-ext/build/chrome-mv3-dev/`.
+ - **Firefox**: Visit `about:debugging`, select **This Firefox**, click **Load Temporary Add-on**, and select `manifest.json` in `packages/flagon-userale-ext/build/firefox-mv3-dev/`.
---
## Building for Production
-To create a production build:
-
```bash
-pnpm build
+pnpm --dir packages/flagon-userale-ext build
```
-This outputs a zipped production bundle.
-
---
## Extension Options
-After installing the extension, click the icon in your browser toolbar to open the **Options** page. From here you can configure:
-- **Logging Endpoint** – Where logs will be sent
-- **URL allowlist** – A regex of URL's to allow logging on
-- **OAuth Credentials** – Details of an OAuth login to authenticate with the logging endpoint.
+Click the UserALE icon in your browser toolbar (or open the extension **Options** page) to configure:
+
+- **Logging Endpoint URL**: Target HTTP(S), WebSocket (`ws://` / `wss://`), or OpenTelemetry Collector (`http://localhost:4318/v1/logs`) URL.
+- **Exporter Format**: Choose between `UserALE JSON (Default)`, `OpenTelemetry (OTLP/JSON)`, or `Protobuf v1 JSON (LogBatch)`.
+- **URL Allowlist**: Regular expression controlling which pages are instrumented.
+- **User ID & Tool Metadata**: Optional user identifier, tool name, and version.
+- **HTTP Basic Auth**: Optional username and password encoded as a `Basic` `Authorization` header.
+- **Custom HTTP Headers (JSON)**: Optional JSON object of custom headers (e.g. `{"x-api-key": "..."}`) attached to every export request.
+- **OAuth 2.0 PKCE Credentials**: Optional OAuth provider configuration for token-authenticated endpoints.
---
## Resources
-- 📚 [UserALE Documentation](https://flagon.apache.org/userale/)
-
+- 📚 [UserALE Documentation](https://flagon.apache.org/docs/useralejs/)
+- 🔭 [OpenTelemetry Demo Stack](../../example/otel-demo/README.md)
diff --git a/site/_docs/contributing.md b/site/_docs/contributing.md
index 3e4951c..50b659b 100644
--- a/site/_docs/contributing.md
+++ b/site/_docs/contributing.md
@@ -4,28 +4,73 @@ component: flagon
permalink: /docs/contributing/
---
-First, thank you for contributing to Apache Flagon!
+First, thank you for contributing to **Apache Flagon**!
-There are certain procedures that must be followed for all contributions. These procedures are necessary to allow us to allocate resources for reviewing and testing your contribution, as well as to communicate effectively with you during the review process.
+All Apache Flagon components (**UserALE.js**, **UserALE Browser Extension**, **Distill**, **Protobuf schemas**, and the **Project Website**) are maintained in the unified [`apache/flagon`](https://github.com/apache/flagon) monorepo.
-See individual product guides for product-specific information on getting started as a contributor, setup and testing, code style, etc.
+---
+
+### 1. Open or Claim an Issue on GitHub
+
+All changes to Apache Flagon should correspond to a [GitHub Issue](https://github.com/apache/flagon/issues) so work can be discussed and tracked transparently.
+
+---
+
+### 2. Set Up Your Local Development Environment
+
+Clone the repository and create a topic branch from `master`:
+
+```shell
+git clone https://github.com/apache/flagon.git
+cd flagon
+git checkout -b feat/your-feature-name
+```
-To report a bug or other issue, simply create an issue in Giithub, as described in Step 1 below.
+#### Option A: Nix Flake (Recommended)
+If you use [Nix](https://nixos.org/) with flakes enabled, run:
-#### 1. Create an issue in Github
+```shell
+nix develop
+```
- All changes to Apache Flagon must have a corresponding issue in [GitHub](https://github.com/apache?q=flagon) so the change can be properly tracked. If you do not already have an account on Github, you will need to create one before creating your new issue.
+This provides Node.js, `pnpm`, TypeScript, Python 3, `buf`, and `lefthook` git hooks automatically.
-#### 2. Make and test your changes locally
+#### Option B: Manual Toolchain Setup
+- **UserALE (`products/userale`)**: Node.js `>= 20.x` and `pnpm >= 10.x`
+- **Distill (`products/distill`)**: Python `>= 3.8` and `poetry` (or `uv`)
+- **Protobuf (`proto/`)**: [`buf` CLI](https://buf.build/docs/installation)
- The Apache Flagon source code is maintained in [several git repositories](https://gitbox.apache.org/repos/asf?s=flagon) hosted by Apache. These repositories are mirrored and more easily available [on GitHub](https://github.com/apache?q=flagon). To make your changes, fork the appropriate GitHub repository and make commits to a topic branch in your fork. Commits should be made in logical units:
- ```shell
- git commit -S -m "[FLAGON-123] High-level message describing the changes."
- ```
- Note that your commits should be verifiable; each commit should be verified with a PGP certificate. The easiest way to do this is to create a PGP signature with [GnuPG](https://www.gnupg.org/), register it with you GitHub account and [configure git](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) to use this signature when committing.
-
- Avoid commits which cover multiple, distinct goals that could (and should) be handled separately. If you do not already have an account on JIRA, you will need to create one to claim an issue, discuss development, or report results.
+---
+
+### 3. Build & Test Your Changes
+
+#### UserALE.js & Browser Extension (`products/userale`)
+```shell
+cd products/userale
+pnpm install
+pnpm build
+pnpm --dir packages/flagon-userale test
+pnpm format
+pnpm lint
+```
+
+#### Distill (`products/distill`)
+```shell
+cd products/distill
+poetry install --all-groups
+poetry run pytest
+```
+
+#### Protobuf Schemas (`proto/`)
+```shell
+buf lint
+buf generate
+```
+
+---
-#### 3. Submit your changes via a pull request on GitHub
+### 4. Submit a Pull Request on GitHub
- Once your changes are ready, submit them by creating a pull request for the corresponding topic branch you created when you began working on your changes. The core team will then review your changes and, if they pass review, your changes will be merged into the primary Apache-hosted repos.
+1. Ensure every new source file includes the standard Apache-2.0 license header.
+2. Commit your changes in logical units with pre-commit hooks enabled (do not pass `--no-verify`).
+3. Open a Pull Request against [`apache/flagon`](https://github.com/apache/flagon/pulls) referencing the issue number (e.g. `Closes #123`).
diff --git a/site/_docs/distill/index.md b/site/_docs/distill/index.md
index d4ea146..d403d7a 100644
--- a/site/_docs/distill/index.md
+++ b/site/_docs/distill/index.md
@@ -1,10 +1,64 @@
---
-title: Installation
+title: Getting Started & API Reference
component: distill
permalink: /docs/distill/
priority: 0
---
-### Installing Apache Distill
+**Apache Flagon Distill** (`flagon-distill`) is a Python behavioral analytics framework for validating, segmenting, and modeling frontend user interaction logs generated by [Apache Flagon UserALE.js]({{ '/docs/useralejs' | prepend: site.baseurl }}).
-#### This project is currently being refactored. If you are interested in Distill, join the discussion on our [dev list](mailto:dev-subscribe@flagon.incubator.apache.org).
\ No newline at end of file
+---
+
+## Installation
+
+### From PyPI
+```bash
+pip install flagon-distill
+```
+
+### From Source (`apache/flagon` Monorepo)
+```bash
+git clone https://github.com/apache/flagon.git
+cd flagon/products/distill
+poetry install --all-groups
+poetry run pytest
+```
+
+---
+
+## Multi-Format UserALE Schema Ingestion (`distill.schemas.userale`)
+
+Distill's `UserAleSchema` and `extract_logs` utilities automatically normalize and validate logs across all three wire formats emitted by UserALE.js:
+
+1. **UserALE Flat JSON (`v2.3` / `v2.4` / `v2.5`)**: Standard flat JSON log objects (`raw`, `interval`, and `custom` log types).
+2. **Protobuf v1 Canonical JSON (`LogBatch`)**: Nested `apache.flagon.userale.v1.LogBatch` payloads produced by `ProtoJsonExporter`.
+3. **OpenTelemetry OTLP/JSON (`ExportLogsServiceRequest`)**: `resourceLogs[].scopeLogs[].logRecords[]` payloads produced by `OtlpJsonExporter`.
+
+```python
+from distill.process.log import Log
+from distill.schemas.userale import UserAleSchema, extract_logs, normalize_userale_dict
+
+# Extract and normalize logs from a flat list, Proto-JSON LogBatch, or OTLP/JSON payload:
+normalized_docs = extract_logs(payload)
+validated_logs = [UserAleSchema(**doc) for doc in normalized_docs]
+
+# Or initialize a Distill Log container directly:
+log = Log(data=payload, schema=UserAleSchema)
+```
+
+---
+
+## Core Modules & API Reference
+
+| Module | Key Classes & Functions | Description |
+|---|---|---|
+| `distill.schemas.userale` | `UserAleSchema`, `extract_logs(payload)`, `normalize_userale_dict(data)` | Pydantic v2 validation and multi-format normalization for UserALE flat JSON, Protobuf v1 `LogBatch`, and OpenTelemetry OTLP/JSON logs. |
+| `distill.process.log` | `Log` | Container for validated interaction streams, indexing by timestamp/PKSUID and supporting DataFrame/dict conversion. |
+| `distill.segmentation.segment` | `Segment`, `Segments`, `create_segment(...)`, `detect_segments(...)` | Time-window, event-driven, and predicate-based slicing of user sessions. |
+| `distill.analytics.graph` | `graph`, `funnel`, `sankey` | Directed behavioral clickstream graphs, conversion funnels, and path visualizations powered by NetworkX and Plotly. |
+
+---
+
+## Contributing
+
+Contributions are welcome! See the [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file
diff --git a/site/_docs/useralejs/API.md b/site/_docs/useralejs/API.md
index d8010c6..6d2f246 100644
--- a/site/_docs/useralejs/API.md
+++ b/site/_docs/useralejs/API.md
@@ -5,99 +5,174 @@ permalink: /docs/useralejs/API/
priority: 2
---
-[Apache UserALE.js](https://github.com/apache/flagon-useralejs) features a robust API that allows you
-customize official Apache Flagon UserALE.js [builds](https://github.com/apache/flagon-useralejs/tree/master/build) page-by-page to suite your needs.
+[Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale) provides a comprehensive JavaScript and TypeScript API for runtime configuration, custom log enrichment/filtering, header injection, and pluggable wire exporters.
+---
-# API Usage
-
-To invoke the API, simply add additional javascript code blocks under your UserALE.js script tag.
+## Core Lifecycle & Configuration API
-Don't forget to add UserALE.js to your project!
-
-It can be deployed as either script-tag, NPM module, or via CDN! For in-depth example, see our [Getting Started Guide]({{ '/docs/useralejs' | prepend: site.baseurl }}) or source [README.md](https://github.com/apache/flagon-useralejs#usage).
+### `options(newConfig?: Partial<Settings.Config>): Settings.Config`
-Also, check out our[example test utility](https://github.com/apache/flagon-useralejs/tree/master/example) so that you can experiment with logs in your terminal!
+Reads or shallow-merges runtime configuration options. Calling `options()` with no arguments returns the current configuration object.
-The following API examples use script-tag deployment methods for illustration:
+```typescript
+import * as userale from 'flagon-userale';
-```html
-<head>
- <title>UserAleJS - Example Page</title>
- <!--
- Add the UserALE.js script tag to the top of your doc
- -->
- <script src="/path/to/userale-2.4.0.min.js" data-url="http://yourLoggingUrl"></script>
+userale.options({
+ url: 'https://collector.example.com/v1/logs',
+ userId: 'user-42',
+ toolName: 'checkout-ui',
+ toolVersion: '2.5.0',
+ exporterFormat: 'otlp-json',
+});
```
-## `options()`
+### `start(): void` & `stop(): void`
-`options` allows you to modify UserALE.js script tag parameters. This can be particularly useful if need to pass additional data about users from your page into the logging parameters, like username and sessionID. You can build variables constructors that pass data from form data or browser storage to UserALE's paramters.
+Starts or halts event capturing and log transmission. Particularly useful when `autostart: false` (`data-autostart="false"`) is configured until user consent or authentication completes.
-```html
-<script type="text/javascript">
- const changeMe = "me";
- window.userale.options({
- "userId": changeMe,
- "version": "2.4.0",
- "sessionID": "4"
- })
-</script>
+```typescript
+userale.options({ autostart: false });
+// Later, after user consent:
+userale.start();
```
-## `addCallbacks()`
-
-`addCallbacks` allows you to modify UserALE.js logs via callback functions. Callback functions are passed the log to be modifyed, and the corresponding event if one exists. The value returned from the calback is used as the new log, except when a falsey value is returned. In the case of a falsey value, the log is dropped. Callbacks are executed in the order they are submitted. Pass multiple callbacks with an object of functions. The following example shows how to both filter unwanted eve [...]
-
-```html
-<script type="text/javascript">
- window.userale.addCallbacks({
- filter(log) {
- var type_array = [
- "mouseup",
- "mouseover",
- "mousedown",
- "keydown",
- "dblclick",
- "blur",
- "focus",
- "input",
- "wheel",
- ];
- var logType_array = ["interval"];
- if (type_array.includes(log.type) || logType_array.includes(log.logType)) {
- return false;
- }
- return log;
- },
- addCustomFields(log, event) {
- var targetsForLabels = ["button#test_button"];
- if (targetsForLabels.includes(log.target)) {
- return Object.assign({}, log, { CustomLabel: "Click me!" });
- } else {
- return log;
- }
+### `version: string`
+
+Exports the current `flagon-userale` package version string.
+
+---
+
+## Custom Logging & Filtering Callbacks
+
+### `addCallbacks(callbacks: Record<string, CallableFunction>): boolean`
+
+Registers one or more named callback functions executed sequentially on every packaged log (`(log, event) => log | false`). Returning `false` (or any falsy value) drops the log from the queue; returning a modified log object replaces it.
+
+```typescript
+userale.addCallbacks({
+ dropIntervalLogs(log) {
+ if (log.logType === 'interval') {
+ return false;
}
- });
-</script>
+ return log;
+ },
+ attachExperimentTag(log, event) {
+ return {
+ ...log,
+ experimentVariant: 'checkout-v2',
+ };
+ },
+});
+```
+
+### `removeCallbacks(targetKeys: string[]): void`
+
+Removes registered callbacks by name:
+
+```typescript
+userale.removeCallbacks(['dropIntervalLogs']);
+```
+
+### `log(customLog: Logging.CustomLog): boolean`
+
+Appends a custom log object directly onto the UserALE queue:
+
+```typescript
+userale.log({
+ target: 'checkout-funnel',
+ type: 'step_complete',
+ logType: 'custom',
+ userAction: false,
+ details: { step: 2, cartTotal: 149.0 },
+});
+```
+
+### `packageLog(e: Event, detailFcn?: Logging.DynamicDetailFunction): boolean` & `packageCustomLog(customLogs: Logging.CustomLog, detailFcn: Logging.DynamicDetailFunction, userAction: boolean): boolean`
+
+Packages a standard DOM `Event` or custom metadata object with full UserALE page/session/browser context and enqueues it.
+
+### `details(config: Configuration, type: Events.AllowedEvents): ((e: Event) => Record<string, unknown>) | null`
+
+Returns the default detail-extractor function for a given DOM event type (`click`, `keydown`, `input`, `change`, `wheel`, etc.). Note that `<input type="password">` values are automatically masked (`"********"`).
+
+### `getSelector(ele: EventTarget): string` & `buildPath(e: Event): string[]`
+
+Utility helpers that compute a CSS-like element selector (`tag#id.class`) and full DOM ancestry path array from a DOM `Event`.
+
+---
+
+## Authentication & Custom HTTP Headers
+
+All HTTP-based exporters (`HttpJsonExporter`, `ProtoJsonExporter`, `OtlpJsonExporter`) construct headers through `buildHeaders(config)`, applying them to both interval `fetch()` requests and `pagehide` (`fetch(..., { keepalive: true })`) flushes.
+
+### `registerAuthCallback(callback: () => string): boolean`
+
+Registers a dynamic provider returning the `Authorization` header value on each batch export:
+
+```typescript
+userale.registerAuthCallback(() => `Bearer ${getCurrentJwtToken()}`);
+```
+
+### `registerHeadersCallback(callback: () => Record<string, string>): boolean`
+
+Registers a dynamic provider returning arbitrary HTTP headers on each batch export:
+
+```typescript
+userale.registerHeadersCallback(() => ({
+ 'x-tenant-id': 'acme-corp',
+ 'x-request-source': 'userale-web',
+}));
```
-## `removeCallbacks()`
+Static headers can also be configured via `userale.options({ authHeader: 'Bearer ...', apiKey: '...', headers: { ... } })`.
-`removeCallbacks` removes a list of callback function that was added by addCallbacks() by name. The following example will remove the filter from the addCallbacks() example
+---
+
+## Pluggable Wire Exporters (`LogExporter`)
+
+UserALE decouples in-memory event packaging from wire serialization and transport via the `LogExporter` interface:
-```html
-<script type="text/javascript">
- window.userale.removeCallbacks(["filter"]);
-</script>
+```typescript
+export interface LogExporter {
+ export(logs: Array<Logging.Log>, options?: ExportOptions): Promise<ExportResult>;
+ flush?(): Promise<void>;
+ shutdown?(): Promise<void>;
+}
```
-Test these examples out in our UserALE.js [example page](https://github.com/apache/flagon-useralejs/blob/master/example/index.html) test utility!
+### Built-In Exporters & Helpers
+
+| Exported Symbol | Description |
+|---|---|
+| `registerExporter(exporter: LogExporter)` | Registers a custom `LogExporter` instance and switches the active sender to use it immediately. |
+| `createExporter(config)` | Factory returning the active `LogExporter` for a `Configuration` instance. |
+| `HttpJsonExporter` | Default exporter sending flat UserALE JSON arrays over `fetch` (with 48 KiB chunked `keepalive: true` on `pagehide`). |
+| `WebSocketExporter` | Streams log batches over a persistent `WebSocket` (`ws://` / `wss://`). |
+| `ProtoJsonExporter` / `toProtoLogBatch(logs)` | Serializes logs into canonical Protobuf v1 JSON (`apache.flagon.userale.v1.LogBatch`) matching `proto/apache/flagon/userale/v1/log.proto`. |
+| `OtlpJsonExporter` / `toOtlpExportLogsServiceRequest(logs, config)` | Serializes logs into OpenTelemetry `ExportLogsServiceRequest` JSON with `Resource` hoisting, `BigInt` nanosecond timestamps, `eventName`, structured `kvlistValue` body, and W3C `traceId`/`spanId` correlation. |
+| `OtelSdkExporter` | Bridges UserALE logs directly into an existing OpenTelemetry JS `@opentelemetry/api-logs` `Logger` via `logger.emit(...)`. |
+| `resolveOtlpLogsEndpoint(endpoint)` | Normalizes an OTLP base URL by appending `/v1/logs` when the URL path is empty or `/`. |
+| `toUnixNanoString(clientTimeMs, microTime)` | Converts millisecond + sub-millisecond timestamps to a 64-bit nanosecond string using `BigInt`. |
+
+### Example: Correlating UserALE Events with W3C Distributed Traces
+
+```typescript
+import * as userale from 'flagon-userale';
+
+userale.options({
+ exporterFormat: 'otlp-json',
+ otlpEndpoint: 'http://localhost:4318/v1/logs',
+ toolName: 'storefront-web',
+ traceContext: () => ({
+ traceId: currentActiveSpan.traceId, // 32-hex W3C trace-id
+ spanId: currentActiveSpan.spanId, // 16-hex W3C span-id
+ }),
+});
+```
-See more [UserALE.js API Examples]()
+---
## Contributing
-Contributions are welcome! Simply [submit an issue](https://github.com/apache/flagon-useralejs/issues) for problems
-you encounter or a pull request for your feature or bug fix. The core team will review it and work with you to
-incorporate it into UserALE.js. We also love Pull Requests!
\ No newline at end of file
+Contributions are welcome! See our [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file
diff --git a/site/_docs/useralejs/build.md b/site/_docs/useralejs/build.md
index 2fe8dc0..6e080ec 100644
--- a/site/_docs/useralejs/build.md
+++ b/site/_docs/useralejs/build.md
@@ -5,116 +5,85 @@ permalink: /docs/useralejs/build/
priority: 1
---
-To build [Apache UserALE.js](https://github.com/apache/flagon-useralejs/tree/master), you will need to clone our repo and install [NPM and Node.js](https://nodejs.org/) and [Go](https://go.dev/doc/install).
+To build and test [Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale), clone the `apache/flagon` monorepo and ensure you have **Node.js (`>= 20.x`)** and **`pnpm` (`>= 10.x`)** installed (or run `nix develop` at the repository root).
-UserALE.js utilizes NPM for package and dependency management. Execute the following to install dependencies.
-```
-#install required packages
-npm install
+```bash
+git clone https://github.com/apache/flagon.git
+cd flagon/products/userale
-#review major dependencies
-npm ls --depth=0
+# Install workspace dependencies
+pnpm install
```
+---
+
## Build
-UserALE.js uses Rollup to manage build tasking. These tasks are executed through our build script; use the following to execute:
+UserALE.js is written in TypeScript and bundled with [`tsup`](https://tsup.egoist.dev/) (producing both ES module `build/main.mjs` and script-tag IIFE `build/main.global.js` bundles alongside TypeScript declarations `build/main.d.ts`), while the browser extension (`flagon-userale-ext`) is built with [Plasmo](https://docs.plasmo.com/).
+```bash
+# Build all workspace packages (flagon-userale and flagon-userale-ext)
+pnpm build
+
+# Or build only flagon-userale
+pnpm --dir packages/flagon-userale build
```
-#Build UserALE.js
-npm run build
+
+If you modify [`proto/apache/flagon/userale/v1/log.proto`](https://github.com/apache/flagon/blob/master/proto/apache/flagon/userale/v1/log.proto), regenerate the TypeScript Protobuf contract definitions from the repository root:
+
+```bash
+buf lint
+buf generate
```
+---
+
## Test
-UserALE.js uses Jest and Playwright to execute tests, which are integrated into our build pipeline. See ``package.json`` for full script options.
+UserALE.js uses **Jest** (with `jsdom` and `@bufbuild/protobuf` contract tests) for unit testing and **Playwright** for end-to-end browser and extension testing.
-To run tests:
-```
-npm run test
+### Run Unit & Schema Contract Tests (`Jest`)
+
+```bash
+pnpm --dir packages/flagon-userale test
```
-... you'll see something like:
+
+Example output:
+
+```text
+PASS test/spec/auth.spec.ts
+PASS test/spec/headers.spec.ts
+PASS test/spec/getInitialSettings.spec.ts
+PASS test/spec/configure.spec.ts
+PASS test/spec/attachHandlers.spec.ts
+PASS test/spec/sendLogs.spec.ts
+PASS test/spec/exporters.spec.ts
+PASS test/spec/packageLogs.spec.ts
+PASS test/spec/main.spec.ts
+
+Test Suites: 9 passed, 9 total
+Tests: 1 skipped, 103 passed, 104 total
```
-...
- attachHandlers
- ✓ attaches all the event handlers without duplicates
- ✓ debounces bufferedEvents (505ms)
- defineDetails
- - configures high detail events correctly
-
- configure
- ✓ merges new configs into main config object
- ✓ includes a userid if present in the window.location
- getUserIdFromParams
- ✓ fetches userId from URL params
- ✓ returns null if no matching param
-
- getInitialSettings
- timeStampScale
- ✓ no event.timestamp
- ✓ zero
- ✓ epoch milliseconds
- ✓ epoch microseconds
- ✓ performance navigation time
- getInitialSettings
- ✓ fetches all settings from a script tag (122ms)
- ✓ grabs user id from params
-
- Userale API
- ✓ provides configs
- ✓ edits configs
- ✓ starts + stops (214ms)
- ✓ sends custom logs
-
- packageLogs
- setLogFilter
- ✓ assigns the handler to the provided value
- ✓ allows the handler to be nulled
- setLogMapper
- ✓ assigns the handler to the provided value
- ✓ allows the handler to be nulled
- packageLog
- ✓ only executes if on
- ✓ calls detailFcn with the event as an argument if provided
- ✓ packages logs
- ✓ filters logs when a handler is assigned and returns false
- ✓ assigns logs to the mapper's return value if a handler is assigned
- ✓ does not call the map handler if the log is filtered out
- ✓ does not attempt to call a non-function filter/mapper
- extractTimeFields
- ✓ returns the millisecond and microsecond portions of a timestamp
- ✓ sets micro to 0 when no decimal is present
- ✓ always returns an object
- getLocation
- ✓ returns event page location
- ✓ calculates page location if unavailable
- ✓ fails to null
- selectorizePath
- ✓ returns a new array of the same length provided
- getSelector
- ✓ builds a selector
- ✓ identifies window
- ✓ handles a non-null unknown value
- buildPath
- ✓ builds a path
- ✓ defaults to path if available
-
- sendLogs
- ✓ sends logs on an interval
- ✓ does not send logs if the config is off
- ✓ sends logs on page exit with navigator
- ✓ sends logs on page exit without navigator
- ✓ does not send logs on page exit if config is off
-
-
- 45 passing (954ms)
- 1 pending
+
+### Run End-to-End Tests (`Playwright`)
+
+From `products/userale`:
+
+```bash
+pnpm test
```
-Followed by e2e tests via playwright that runs in a headed browser. Note that interacting with the headed browser may contaminate test results. Unfortanutely, the browser plugin requires playwright to run with a headed browser at this time. Any failing tests will also be logged in the terminal. If there are failing tests, please consider [opening an issue in Github](https://github.com/apache/flagon-useralejs/issues).
+---
-Further instructions can be found in our [README](https://github.com/apache/flagon-useralejs/blob/master/README.md)
+## Code Formatting & Linting
+
+```bash
+pnpm format
+pnpm lint
+```
+
+---
## Contributing
-Contributions are welcome! See the [contribution guide](https://github.com/apache/flagon-useralejs/blob/master/CONTRIBUTING.md) for more information.
\ No newline at end of file
+Contributions are welcome! See the [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) for more information.
\ No newline at end of file
diff --git a/site/_docs/useralejs/dataschema.md b/site/_docs/useralejs/dataschema.md
index ce5ed2b..b3a7b0a 100644
--- a/site/_docs/useralejs/dataschema.md
+++ b/site/_docs/useralejs/dataschema.md
@@ -5,249 +5,120 @@ permalink: /docs/useralejs/dataschema/
priority: 4
---
-This guide describes the structure of content of [Apache UserALE.js](https://github.com/apache/flagon-useralejs)
-logs, including:
- * which behaviors UserALE.js listens to;
- * which meta data UserALE.js attaches to logs, for context:
- * different types of UserALE.js logs, and their usage.
+This guide describes the structure of [Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale) logs, including:
+- The in-memory and default flat JSON schema (`'userale-json'`)
+- The OpenTelemetry OTLP/JSON export schema (`'otlp-json'`)
+- The Protobuf v1 canonical JSON export schema (`'proto-json'`, [`proto/apache/flagon/userale/v1/log.proto`](https://github.com/apache/flagon/blob/master/proto/apache/flagon/userale/v1/log.proto))
-## UserALE.js Logs
+All three formats are natively supported by **Apache Flagon Distill** (`distill.schemas.userale.extract_logs` and `UserAleSchema`).
-UserALE.js logs, by default, contain a lot more data than other business analytics products because it is powerful enough for *workflow analysis*.
-
-Workflow analysis is about understanding the complex interdependency between elements and behaviors within and across
-pages. Workflow analysis at this level is really crucial for understanding:
-* how people think about the information on your site or app
-* understanding efficiency in information discovery
-* and for gauging how novices explore pages compared to experts or superusers.
+---
-In order to do workflow analysis, UserALE.js generates granular data about interactions with page elements.
+## 1. Standard UserALE Flat JSON Schema (`exporterFormat: 'userale-json'`)
-This is a standard UserALE.js log from our [example test utility](https://github.com/apache/flagon-useralejs/blob/master/example/index.html):
+By default, UserALE.js packages each interaction as a flat JSON object so `userale.addCallbacks(...)` filters and mappers can inspect or transform fields with zero nesting overhead.
+### Raw Interaction Log (`logType: 'raw'`)
-```html
+```json
{
- target: 'button#test_button',
- path: [
- 'button#test_button',
- 'div.container',
- 'body',
- 'html',
- '#document',
- 'Window'
+ "target": "button#test_button",
+ "path": [
+ "button#test_button",
+ "div.container",
+ "body",
+ "html",
+ "#document",
+ "Window"
],
- pageUrl: 'http://localhost:8000/',
- pageTitle: 'UserALE - Example Page',
- pageReferrer: '',
- browser: { browser: 'firefox', version: '131.0.0' },
- clientTime: 1729658103797,
- microTime: 0,
- location: { x: 66, y: 220 },
- scrnRes: { width: 1920, height: 915 },
- type: 'click',
- logType: 'raw',
- userAction: true,
- details: { clicks: 1, ctrl: false, alt: false, shift: false, meta: false },
- userId: 'me',
- toolVersion: '2.3.0',
- toolName: 'Apache UserALE Example (Custom)',
- useraleVersion: '2.4.0',
- sessionId: 'session_1729657762020',
- httpSessionId: '057109eed2c0b107476731f201b6d63d',
- browserSessionId: null,
- attributes: { id: 'test_button' },
- style: {}
+ "pageUrl": "http://localhost:8000/",
+ "pageTitle": "UserALE - Example Page",
+ "pageReferrer": "",
+ "browser": { "browser": "firefox", "version": "131.0.0" },
+ "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0",
+ "clientTime": 1729658103797,
+ "microTime": 0.42,
+ "location": { "x": 66, "y": 220 },
+ "scrnRes": { "width": 1920, "height": 915 },
+ "type": "click",
+ "logType": "raw",
+ "userAction": true,
+ "details": { "clicks": 1, "ctrl": false, "alt": false, "shift": false, "meta": false },
+ "userId": "me",
+ "toolVersion": "2.4.0",
+ "toolName": "Apache UserALE Example",
+ "useraleVersion": "2.4.0",
+ "sessionId": "session_1729657762020",
+ "httpSessionId": "057109eed2c0b107476731f201b6d63d",
+ "browserSessionId": null,
+ "attributes": { "id": "test_button" },
+ "style": {}
}
```
+### Interval Summary Log (`logType: 'interval'`)
-It's a lot, but it makes sense in context.
-
-### UserALE.js Event Tracking
-
-The fields dentify what users do and what they interact with. UserALE.js gets this information from anywhere on a page/app without using a ton of hooks.
-
-
-```html
- type: 'click', [this field documents the kind of user behavior observed]
- target: 'button#test_button', [this field documents the site/app element a user interacted with]
- path: [ {this is the DOM path to the "target" element, which is critical for disambiguation between elements}
- 'button#test_button',
- 'div.container',
- 'body',
- 'html',
- '#document',
- 'Window'
- ]
-
-```
-
-
-You may want to specially label some of these fields and important elements that. See our guide on [the UserALE.js API]({{ '/docs/useralejs/API' | prepend: site.baseurl }}).
-
-For more information about the kinds of behaviors (event classes) we collect, check out our guide for
-[modifying UserALE.js source code]({{ '/docs/useralejs/modifying' | prepend: site.baseurl }}).
-
-### Temporal Handling in UserALE.js
-
-Time is a very important concept in UserALE.js. Below are the fields related to temporal data capture:
-
-
-```html
- clientTime: 1729658103797, [UserALE.js collects time from the client when a log is generated]
- microTime: 0, [When a log is wrtten, we write microtime too, not every database parses it]
-```
-
-
-### UserALE.js Meta-Data
-
-You want context? We have all of the things for all of your events:
-
-#### Event Context
-
-Every event has a little nuance, UserALE.js gets it all.
-
-
-```html
- details: {
- clicks: 1,
- ctrl: false,
- alt: false,
- shift: false,
- meta: false
- },
-```
-
-You'll get location of the cursor two for mouseover and click data, too!
-
-
-```html
- location: {
- x: 66,
- y: 220
- }
-```
-
-
-[UserALE.js' WebExtension](https://github.com/apache/flagon-useralejs/tree/master/build) is deployed directly
-into your browser, as a result you'll get browser events.
-
-Examples are below,you'll see our details field has some pretty useful data from the browser.
-
-
-```html
- details: {
- active: true,
- audible: false,
- autoDiscardable: true,
- discarded: false,
- favIconUrl: 'https://apache.org/favicons/favicon-32x32.png',
- groupId: -1,
- height: 914,
- highlighted: true,
- id: 1852931762,
- incognito: false,
- index: 4,
- lastAccessed: 1729720562214.495,
- openerTabId: 1852931752,
- pinned: false,
- selected: true,
- status: 'complete',
- title: 'Welcome to The Apache Software Foundation!',
- url: 'https://apache.org/',
- width: 1912,
- windowId: 1852931606
- }
-```
-
-
-#### Page/App Context
-
-If a user clicks on a thing and you know nothing about that thing, you're toast!
-
-UserALE.js gives a lot context about your page or app. Some are automatically grabbed from the page, some are
-configurable through our [script tag]({{ '/docs/useralejs' | prepend: site.baseurl }})
-
-
-```html
- "toolVersion": "2.4.0", [you can give a semantic versioning of your page or tool]
- "pageReferrer": "", [grabbed from the page, from which page did your users find your page/app]
- "toolName": "Apache UserALE.js Example", [give your tool an alis through our script tag]
- "pageTitle": "UserAleJS - Example Page", [page title grabbed from page]
- "pageUrl": "file:///.../Apache_Flagon/test/flagon-useralejs/example/index.html"
-```
-
-
-To find out more and how to change the context attached to event logs, check out our
-[modifying UserALE.js source code]({{ '/docs/useralejs/modifying' | prepend: site.baseurl }}) guide.
+High-frequency events (`mouseover`, `scroll`, etc.) are aggregated over `resolution` ms windows into `interval` logs with `startTime`, `endTime`, `duration`, and `count`:
-#### User Context
-
-UserALE.js has its own methods for gathering user context, and it can integrate with other methods you might have
-(e.g., cookies)
-
-
-```html
- sessionId: 'session_1729657762020', [This session ID is unique to a specific page visit (deprecated)]
- httpSessionId: '057109eed2c0b107476731f201b6d63d', [This ID is unique to the page session]
- browserSessionId: null, [This ID is unique to the browser session when using the plugin]
- userId: 'me', [You can also pass a unique userId]
+```json
+{
+ "target": "div.feed",
+ "path": ["div.feed", "body", "html"],
+ "pageUrl": "https://apache.org/",
+ "pageTitle": "Welcome to The Apache Software Foundation!",
+ "pageReferrer": "",
+ "browser": { "browser": "chrome", "version": "125.0.0" },
+ "count": 14,
+ "duration": 480,
+ "startTime": 1729720562000,
+ "endTime": 1729720562480,
+ "type": "mouseover",
+ "logType": "interval",
+ "targetChange": true,
+ "typeChange": false,
+ "userAction": false,
+ "userId": "me",
+ "toolVersion": "2.4.0",
+ "toolName": "Apache UserALE Example",
+ "useraleVersion": "2.4.0",
+ "sessionId": "session_1727742067433",
+ "httpSessionId": "298926d0ffe8085dc0f7c421dbad9c3a",
+ "browserSessionId": "d43400275c1f4f04f1ce74da7d0cd3c8"
+}
```
+---
-To see how to pass in a userId from a url, check out our [script tag docs]({{ '/docs/useralejs' | prepend: site.baseurl }}).
-
-Other things you might want to do (e.g., cookies) can also be passed to userId via [the UserALE.js API]({{ '/docs/useralejs/API' | prepend: site.baseurl }}).
-
-## UserALE.js Log Types
+## 2. OpenTelemetry OTLP/JSON Schema (`exporterFormat: 'otlp-json'`)
+
+When `exporterFormat: 'otlp-json'` (or `data-exporter="otlp"`) is enabled, `OtlpJsonExporter` serializes each batch into an OpenTelemetry `ExportLogsServiceRequest` JSON payload:
+
+- **Resource Attributes (`resourceLogs[].resource.attributes`)**: Constant session/device metadata is hoisted once per batch:
+ - `service.name` $\leftarrow$ `toolName` (defaults to `"unknown_service:userale"`)
+ - `service.version` $\leftarrow$ `toolVersion`
+ - `telemetry.sdk.name` $\leftarrow$ `"apache-flagon-userale"`
+ - `telemetry.sdk.language` $\leftarrow$ `"webjs"`
+ - `telemetry.sdk.version` $\leftarrow$ `useraleVersion`
+ - `user_agent.original` $\leftarrow$ `userAgent`
+ - `browser.name`, `browser.version` $\leftarrow$ `browser`
+ - `session.id` $\leftarrow$ `sessionId`
+ - `flagon.http_session_id`, `flagon.browser_session_id`
+ - `screen.width`, `screen.height` $\leftarrow$ `scrnRes`
+- **LogRecord Fields (`scopeLogs[].logRecords[]`)**:
+ - `timeUnixNano` / `observedTimeUnixNano`: 64-bit nanosecond string computed with `BigInt` (`clientTime * 1_000_000 + subMsNanos`)
+ - `eventName`: `"userale.<logType>.<type>"` (e.g. `"userale.raw.click"`, `"userale.interval.mouseover"`)
+ - `traceId` / `spanId`: Populated from `log.traceId`/`log.spanId` or `config.traceContext()`
+ - `body`: Structured `kvlistValue` representation of `log.details`
+ - `attributes`: `event.name`, `url.full`, `user.id`, `flagon.log_type`, `flagon.event_type`, `flagon.target`, `flagon.path` (`arrayValue`), `flagon.page_title`, `flagon.page_referrer`, `flagon.user_action`, `flagon.location.x`, `flagon.location.y`, `flagon.attributes`, `flagon.style`, and interval metrics (`flagon.count`, `flagon.duration_ms`, `flagon.start_time_ms`, `flagon.end_time_ms`)
-For the most part, UserALE.js logs are uniform. However, there are some variations based on the type of log.
+---
-```
+## 3. Protobuf v1 Canonical JSON Schema (`exporterFormat: 'proto-json'`)
-### Interval Logs
+When `exporterFormat: 'proto-json'` is configured, `ProtoJsonExporter` serializes each batch into canonical Proto3 JSON conforming to [`proto/apache/flagon/userale/v1/log.proto`](https://github.com/apache/flagon/blob/master/proto/apache/flagon/userale/v1/log.proto) (`apache.flagon.userale.v1.LogBatch`), grouping common context into `metadata` (`CommonMetadata`) and using the `raw`, `interval`, or `custom` `oneof` variant on each `UserALELog`.
-Interval logs are a experimental feature, that write long sequences of the same event to windows. They capture high resolution data like 'mouseovers' in time-boxed summary windows.
-
-Interval logs have a start- and end-time instead of a single, discrete time-stamp.
+---
-```html
-{
- target: 'img.img-responsive center-block visible-home', [where were mouseovers observed within the interval]
- path: [
- 'img.img-responsive center-block visible-home',
- 'div.main',
- 'header#main-header.container',
- 'body#home',
- 'html',
- '#document',
- 'Window'
- ],
- pageUrl: 'https://apache.org/',
- pageTitle: 'Welcome to The Apache Software Foundation!',
- pageReferrer: 'https://www.bing.com/',
- browser: { browser: 'edge-chromium', version: '130.0.0' },
- count: 0, [count of disrcete events within the interval]
- duration: 67, [duration of interval in ms]
- startTime: 1729720562466, [begin interval]
- endTime: 1729720562533, [end interval]
- type: 'mouseover',
- logType: 'interval', [as compared to "raw"]
- targetChange: true,
- typeChange: false, [were there any other events during the interval?]
- userAction: false,
- userId: 'pluginUser',
- toolVersion: '2.4.0',
- toolName: 'useralePlugin',
- useraleVersion: '2.4.0',
- sessionId: 'session_1727742067433',
- httpSessionId: '298926d0ffe8085dc0f7c421dbad9c3a',
- browserSessionId: 'd43400275c1f4f04f1ce74da7d0cd3c8'
- }
-```
## Contributing
-Contributions are welcome! Simply [submit an issue](https://github.com/apache/flagon-useralejs/issues) for problems
-you encounter or a pull request for your feature or bug fix. The core team will review it and work with you to
-incorporate it into UserALE.js. We also love Pull Requests!
\ No newline at end of file
+Contributions are welcome! See our [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file
diff --git a/site/_docs/useralejs/index.md b/site/_docs/useralejs/index.md
index 4c2d2f0..8d7d3fb 100644
--- a/site/_docs/useralejs/index.md
+++ b/site/_docs/useralejs/index.md
@@ -5,118 +5,106 @@ permalink: /docs/useralejs/
priority: 0
---
-[Apache UserALE.js](https://github.com/apache/flagon-useralejs) is the UserALE client for DOM and JavaScript-based applications. It automatically attaches event handlers, is configurable through HTML5 data parameters or a JS API, and logs every user interaction on a web page, including rich JS single-page apps. You can choose to include UserALE.js directly in your project to track how users engage with it, or if you're interested in generating logs from any webpage, you can try out our [...]
+[Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale) is the client-side instrumentation library for DOM and JavaScript applications. It automatically attaches event handlers, is configurable through HTML5 `data-*` attributes or a JavaScript/TypeScript API, and logs user interactions across web pages and single-page applications (SPAs) with zero runtime dependencies.
-*Note:* Work on UserALE.js' documentation is ongoing. Contributions are welcome. To get involved, see our [Contributing]({{ '/docs/contributing' | prepend: site.baseurl }}) guide.
-## Include UserALE.js in your project
+---
-To start logging with Apache UserALE.js, you can include our script directly in your project. More details can be found in our [README](https://github.com/apache/flagon-useralejs/blob/master/README.md). You can use our [build process]({{ '/docs/useralejs/build' | prepend: site.baseurl }}) to create our script, use a sample in our [repos](https://github.com/apache/flagon-useralejs/tree/master/build), or pull our scripts from [npm](https://www.npmjs.com/package/flagon-userale).
+## Include UserALE.js in Your Project
-To include UserALE.js in a specific project, you'll need to deploy a version of our minified UserALE.js script in an accessible location into your project:
+Install `flagon-userale` from [npm](https://www.npmjs.com/package/flagon-userale):
-```html
+```bash
npm install flagon-userale
+# or
+pnpm add flagon-userale
```
-### Include UserALE.js as a `module`:
+### 1. ES Module / Bundler Import (`ESM`)
-```html
+```typescript
import * as userale from 'flagon-userale';
-or
-
-const userale = require('flagon-userale');
+userale.options({
+ url: 'http://localhost:8000',
+ toolName: 'my-web-app',
+ toolVersion: '1.0.0',
+ logDetails: true,
+});
```
-Our [webpack example](https://github.com/apache/flagon-useralejs/tree/master/example/webpackUserAleExample) illustrates this use-case.
-
-### Include UserALE.js as a `script-tag`:
+### 2. Script Tag (`IIFE`)
```html
-<script src="./node_modules/flagon-userale/build/userale-2.4.0.min.js"></script>
+<script
+ src="./node_modules/flagon-userale/build/main.global.js"
+ data-url="http://localhost:8000/"
+ data-tool="my-web-app"
+ data-version="1.0.0"
+></script>
```
-Our [script tag example](https://github.com/apache/flagon-useralejs/tree/master/example) illustrates this use-case
+### 3. OpenTelemetry Collector (`OTLP/JSON`)
-If you include UserALE.js as a `script-tag`, consider installing via npm as a development dependency:
+UserALE.js natively exports OpenTelemetry Logs (`ExportLogsServiceRequest` over HTTP/JSON) without requiring `@opentelemetry/*` packages:
```html
-npm install --save-dev flagon-userale
+<script
+ src="./node_modules/flagon-userale/build/main.global.js"
+ data-exporter="otlp"
+ data-otlp-endpoint="http://localhost:4318/v1/logs"
+ data-tool="checkout-web"
+></script>
```
-### Include UserALE.js via a CDN:
+Try the full-stack [OpenTelemetry + Grafana + Loki + Tempo Demo](https://github.com/apache/flagon/tree/master/products/userale/example/otel-demo) with `docker compose up -d`!
-```html
-<script src="https://cdn.jsdelivr.net/npm/flagon-userale@2.4.0/build/userale-2.4.0.min.js"></script>
-```
-
-We also support a [WebExtension](https://github.com/apache/flagon-useralejs/tree/master/src/UserALEWebExtension) that can be added to your browser in developer mode. Follow the link for instructions.
+---
-Once UserALE.js is installed in your project, your application will start generating logs automatically.
+## Configuration Options
+
+All options can be configured via HTML `data-*` attributes on the `<script>` tag or programmatically via `userale.options({...})`:
+
+| JS Option | Script `data-*` Attribute | Description | Default |
+|---|---|---|---|
+| `url` | `data-url` | Logging endpoint URL (`http://`, `https://`, `ws://`, `wss://`) | `'http://localhost:8000'` |
+| `autostart` | `data-autostart` | Start capturing and sending logs automatically on load | `true` |
+| `transmitInterval` | `data-interval` | Interval between batch transmit checks (ms) | `5000` |
+| `logCountThreshold` | `data-threshold` | Minimum queued logs required before interval send | `5` |
+| `userId` | `data-user` | User identifier | `null` |
+| `sessionId` | `data-session` | Session identifier | Auto-generated |
+| `toolName` | `data-tool` | Application / tool name (`service.name` in OTLP) | `null` |
+| `toolVersion` / `version` | `data-version` | Application semantic version (`service.version` in OTLP) | `null` |
+| `logDetails` | `data-log-details` | Include detailed event payloads (e.g. key codes, input values; password inputs are automatically masked) | `false` |
+| `resolution` | `data-resolution` | Aggregation interval for high-frequency events (`mouseover`, `scroll`, etc.) in ms | `500` |
+| `userFromParams` | `data-user-from-params` | URL query parameter to extract `userId` from | `null` |
+| `authHeader` | `data-auth` | `Authorization` header string or callback | `null` |
+| `apiKey` | `data-api-key` | Value sent in the `x-api-key` HTTP header | `null` |
+| `exporterFormat` / `exporterType` | `data-exporter` | Wire format: `'userale-json'` (default), `'otlp-json'` (`'otlp'`), `'proto-json'`, or `'ws'` | `'userale-json'` |
+| `otlpEndpoint` | `data-otlp-endpoint` | OpenTelemetry Collector HTTP logs endpoint (automatically appends `/v1/logs` if given a base origin) | `null` |
+| `headers` | — | Custom HTTP headers `Record<string, string>` merged into every request | `null` |
+| `exporter` | — | Custom `LogExporter` instance implementing `export(logs, options)` | `null` |
+| `traceContext` | — | Optional `() => ({ traceId, spanId })` callback for W3C TraceContext correlation | `null` |
+
+See the [UserALE.js API Reference]({{ '/docs/useralejs/API' | prepend: site.baseurl }}) for full method and exporter documentation.
-## Configure UserALE.js
+---
-UserALE.js is designed to be easily configured to fit your use case. In `script-tag` deployment, use HTML data parameters to pass configuration options to the library.
+## Deploying the UserALE.js Browser Extension
-For example, one thing you'll need to do is set the URL (location) of your minified UserALE.js script:
+The [UserALE Browser Extension](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale-ext) is a Manifest V3 extension built with Plasmo that injects UserALE.js into pages matching a configurable URL allowlist.
-```html
-<script src="/path/to/userale-2.4.0.min.js" data-url="http://yourLoggingUrl"></script>
-```
+1. Build the workspace from `products/userale`:
+ ```bash
+ pnpm install
+ pnpm build
+ ```
+2. Load the extension in your browser:
+ - **Chrome / Edge**: Open `chrome://extensions/`, enable **Developer mode**, click **Load unpacked**, and select `products/userale/packages/flagon-userale-ext/build/chrome-mv3-prod/` (or `chrome-mv3-dev/` when running `pnpm dev`).
+ - **Firefox**: Open `about:debugging`, select **This Firefox**, click **Load Temporary Add-on**, and select `manifest.json` in `products/userale/packages/flagon-userale-ext/build/firefox-mv3-prod/`.
+3. Open the extension **Options** page to configure the **Logging Endpoint**, **Exporter Format** (`UserALE JSON`, `OpenTelemetry (OTLP/JSON)`, or `Protobuf v1 JSON`), **URL Allowlist**, **HTTP Basic Auth**, **Custom HTTP Headers (JSON)**, or **OAuth PKCE** credentials.
-The complete list of configurable options is:
-
-| Param | Description | Default |
-|---|---|---|
-| url | Logging URL | http://localhost:8000 |
-| autostart | Should UserALE start on page load | true |
-| transmitInterval | Delay between transmit checks | 5000 (ms) |
-| logCountThreshold | Minimum number of logs to send | 5 |
-| userId | User identifier | null |
-| sessionId | Session identifier | null |
-| version | Application version identifier | null |
-| logDetails | Toggle detailed logs (keys pressed and input/change values) | false |
-| resolution | Delay between instances of high frequency logs (mouseover, scroll, etc.) | 500 (ms) |
-| userFromParams | Query param in the page URL to fetch userId from | null |
-| toolName | Name of tool being logged | null |
-| authHeader | Authorization header to be passed to logging endpoint | null |
-
-**NOTE** These options are also available through our [API]({{ '/docs/useralejs/API' | prepend: site.baseurl }}), which support deploying UserALE.js as an NPM `module.`
-
-See our page on [Build and Test]({{ '/docs/useralejs/build' | prepend: site.baseurl }}) for UserALE.js to learn how to build your own minified version of the UserALE.js script, or you check-out a pre-built script with default parameters in our [repo](https://github.com/apache/flagon-useralejs/tree/master/build).
-
-Use our [example test utility](https://github.com/apache/flagon-useralejs/tree/master/example) to experiment with script tag parameters.
-
-## Deploying the UserALE.js WebExtension
-
-If you're interested in using our Web Extension to log user activity across all pages they visit, try our web extension, which injects UserALE.js into any web page your client(s) navigate to. This option is perfect for testing and research. The web extension builds through the same [build pipeline]({{ '/docs/useralejs/build' | prepend: site.baseurl }}) as UserALE.js. You can also use pre-built extension files in our [repo](https://github.com/apache/flagon-useralejs/tree/master/build).
-
-When you're ready to deploy the UserALE.js web extension, follow the instructions below:
-
-1. Load the web extension into your browser.
- 1. Firefox
- 1. Open Firefox
- 1. Enter about:debugging into the URL bar and press enter
- 1. Check the box to 'Enable add-on debugging'
- 1. Press the 'Load Temporary Add-on' button
- 1. Navigate to the root of the web extension directory (e.g. 'build/UserAleWebExtension')
- 1. Press Open, and confirm that 'User ALE Extension' appears in the list
- 1. You may now navigate to a web page to inject the User ALE script!
- 1. Chrome
- 1. Open Chrome
- 1. Enter chrome://extensions into the URL bar and press enter
- 1. Check the 'Developer mode' box
- 1. Press the 'Load unpacked extension' button
- 1. Navigate to the root of the build directory (e.g. 'build/UserAleWebExtension')
- 1. Press Ok, and confirm that 'UserALE Extension' appears in the list
- 1. You may now navigate to a web page to inject the User ALE script!
-
-You can set options for the web extension in your browser by opening the extensions page, finding the extension, and choosing either "Preferences" for Firefox, or "Options" for Chrome.
-
-You can also find more instructions in our [README](https://github.com/apache/flagon-useralejs/tree/master/src/UserALEWebExtension).
+---
## Contributing
-Contributions are welcome! Simply [submit an issue](https://github.com/apache/flagon-useralejs/issues) for problems
-you encounter or a pull request for your feature or bug fix. The core team will review it and work with you to
-incorporate it into UserALE.js. We also love Pull Requests!
\ No newline at end of file
+Contributions are welcome! See our [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file
diff --git a/site/_docs/useralejs/modifying.md b/site/_docs/useralejs/modifying.md
index 843464d..13122b9 100644
--- a/site/_docs/useralejs/modifying.md
+++ b/site/_docs/useralejs/modifying.md
@@ -5,139 +5,40 @@ permalink: /docs/useralejs/modifying/
priority: 3
---
-[Apache UserALE.js](https://github.com/apache/flagon-useralejs) makes it easy for you to capture *every* user
-behavior on your webpage or application. But, you might not need all that data.
+[Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale) makes it easy to capture and customize behavioral telemetry via the runtime [JavaScript API]({{ '/docs/useralejs/API' | prepend: site.baseurl }}) (`addCallbacks`, `options`, `registerExporter`). When you need deeper changes, the TypeScript source code under `products/userale/packages/flagon-userale/src/` is modular and straightforward to extend.
-UserALE.js also makes it easy for your to manage, and curate your logging scheme.
-
-This guide walks you through how to modify UserALE.js log data and behavior through source code modification.
-
-## Modifying UserALE.js Logs via Source Code Modifications
-
-We're starting with this method because it will give context to others methods, like our script tag and API (below).
-
-UserALE.js src is intuitively abstracted into few different files:
-
- * [attachHandlers](https://github.com/apache/flagon-useralejs/blob/master/src/attachHandlers.js): configure which user behaviors UserALE.js listens for and event handlers here.
- * [configure](https://github.com/apache/flagon-useralejs/blob/master/src/configure.js): set special custom parameters, like passing a unqiue userID into UserALE.js logs.
- * [getInitialSettings](https://github.com/apache/flagon-useralejs/blob/master/src/getInitialSettings.js): modify HTML5 parameters passed through the script tag.
- * [main](https://github.com/apache/flagon-useralejs/blob/master/src/main.js): attaches event handlers and adds logs to queue (probably want to leave alone).
- * [packageLogs](https://github.com/apache/flagon-useralejs/blob/master/src/packageLogs.js): modify to configure how user behaviors are added to data schema, and meta data added from the page.
- * [sendLogs](https://github.com/apache/flagon-useralejs/blob/master/src/sendLogs.js): modify how logs are sent to logging server, etc. (be careful here, too).
-
-### Usage
-
-Most modifications can be accomplished through `attachHandlers` and `packageLogs`.
-
-Once you modify source, you'll need to rebuild UserALE.js scripts and webextension (and should probably run unit tests).
-
-### Modifying Event Handlers: Examples
-
-`attachHandlers` controls which behaviors and what kinds of logs are captured
-
-You can modify some of the events UserALE.js by editing vars in `attacheHandlers`.
-
-
-```javascript
-var events;
-var bufferBools;
-var bufferedEvents;
-/* modify which events are abstracted into interval logs by editing the array below*/
-var intervalEvents = ['click', 'focus', 'blur', 'input', 'change', 'mouseover', 'submit'];
-/* modify which events are pulled from window*/
-var windowEvents = ['load', 'blur', 'focus'];
-```
-
-
-You can modify which events UserALE.js captures in it's 'raw' log stream, by modifying the var `events`:
-
-
-
-```javascript
-/* Comment out event types in the `events` var for granular modification of UserALE.js behavior */
-export function defineDetails(config) {
- // Events list
- // Keys are event types
- // Values are functions that return details object if applicable
- events = {
- 'click' : extractMouseEvent,
- 'dblclick' : extractMouseEvent,
- 'mousedown' : extractMouseEvent,
- 'mouseup' : extractMouseEvent,
- 'focus' : null,
- 'blur' : null,
- 'input' : config.logDetails ? function(e) { return { 'value' : e.target.value }; } : null,
- 'change' : config.logDetails ? function(e) { return { 'value' : e.target.value }; } : null,
- 'dragstart' : null,
- 'dragend' : null,
- 'drag' : null,
- 'drop' : null,
- 'keydown' : config.logDetails ? function(e) { return ...,
- /*'mouseover' : null,*/ #Aliasing the "events" list removes that class from your event stream.
- 'submit' : null
-```
-
-
-You can eliminate the `interval` log stream altogether by modifying the `attachHandlers` configs:
-
-
-```javascript
-export function attachHandlers(config) {
- defineDetails(config);
-
- Object.keys(events).forEach(function(ev) {
- document.addEventListener(ev, function(e) {
- packageLog(e, events[ev]);
- }, true);
- });
-/*Comment out configs for intervalEvents to remove them from your log stream */
- intervalEvents.forEach(function(ev) {
- document.addEventListener(ev, function(e) {
- packageIntervalLog(e);
- }, true);
- });
-```
+---
-### Modifying Meta Data Packaged in Logs: Examples
+## Source Code Architecture (`products/userale/packages/flagon-userale/src/`)
+
+- [`main.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/main.ts): Public module entry point exporting lifecycle functions (`start`, `stop`, `options`, `log`), callback helpers, and exporters.
+- [`getInitialSettings.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/getInitialSettings.ts): Extracts initial settings from `<script data-*>` attributes and browser session storage (with SSR safety guards).
+- [`configure.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/configure.ts): Singleton `Configuration` class managing runtime options.
+- [`attachHandlers.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/attachHandlers.ts): Defines captured DOM/window events, interval buffering, and detail extraction (including automatic masking for `<input type="password">`).
+- [`packageLogs.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/packageLogs.ts): Packages DOM events and custom events into UserALE log objects, executing registered `addCallbacks` functions.
+- [`sendLogs.ts`](https://github.com/apache/flagon/blob/master/products/userale/packages/flagon-userale/src/sendLogs.ts): Manages the batch interval timer and `pagehide` unload flushing, delegating wire serialization and transport to the active `LogExporter`.
+- [`exporters/`](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale/src/exporters):
+ - `types.ts`: `LogExporter`, `ExportOptions`, `ExportResult`, and `ExporterType` interfaces.
+ - `headers.ts`: Unified `buildHeaders(config)` and `chunkLogsForKeepalive` (48 KiB chunking for `fetch(..., { keepalive: true })`).
+ - `http-json.ts`: Default `HttpJsonExporter` (`'userale-json'`).
+ - `otlp-json.ts`: Native OpenTelemetry `OtlpJsonExporter` (`'otlp-json'`).
+ - `otel-sdk.ts`: OpenTelemetry JS Logs API bridge (`OtelSdkExporter`).
+ - `proto-json.ts`: Canonical Protobuf v1 JSON exporter (`ProtoJsonExporter`, `'proto-json'`).
+ - `websocket.ts`: Persistent `WebSocketExporter` (`'ws'`).
-`packageLogs` pulls parameters from the script tag, API, and from window or document and adds them to event logs.
+---
-The easiest way to make these changes is by modifying the var `log`:
+## Rebuilding & Verifying Modifications
+After modifying any file in `products/userale/packages/flagon-userale/src/`, rebuild and run the unit test suite from `products/userale`:
-```javascript
- /* Comment or add fields to be included in your logs*/
- var log = {
- 'target' : getSelector(e.target),
- 'path' : buildPath(e),
- 'pageUrl': window.location.href,
- 'pageTitle': document.title,
- 'pageReferrer': document.referrer,
- 'clientTime' : timeFields.milli,
- 'microTime' : timeFields.micro,
- 'location' : getLocation(e),
- 'type' : e.type,
- 'logType': 'raw',
- 'userAction' : true,
- 'details' : details,
- 'userId' : config.userId,
- 'toolVersion' : config.version,
- 'toolName' : config.toolName,
- 'useraleVersion': config.useraleVersion,
- 'sessionID': config.sessionID
- };
+```bash
+pnpm build
+pnpm --dir packages/flagon-userale test
```
-### Benefits Modifying UserALE.js Logs via src:
- * Full control of your log streams
- * With simple modification to src you can package and build multiple UserALE.js scripts for page-to-page modification
- of log streams.
- * Modify the behavior of our webExtension
- * Ideal in cases where you don't want to integrate UserALE.js into your CD pipeline, just pass around pre-built scripts
- that suite your needs.
+---
## Contributing
-Contributions are welcome! Simply [submit an issue](https://github.com/apache/flagon-useralejs/issues) for problems
-you encounter or a pull request for your feature or bug fix. The core team will review it and work with you to
-incorporate it into UserALE.js. We also love Pull Requests!
\ No newline at end of file
+Contributions are welcome! See our [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file
diff --git a/site/_docs/useralejs/troubleshooting.md b/site/_docs/useralejs/troubleshooting.md
index f362e22..3c0adfd 100644
--- a/site/_docs/useralejs/troubleshooting.md
+++ b/site/_docs/useralejs/troubleshooting.md
@@ -7,12 +7,40 @@ priority: 5
### Enable Cross-Origin Resource Sharing (CORS)
-The logs generated by [Apache UserALE.js](https://github.com/apache/flagon-useralejs) are sent to a logging server which is configured in the script tag or via .js scripts using our API. Your client may send out failure logs because of a CORS (cross-origin resource sharing) issue. You will need to configure your backend server to enable CORS.
+Logs generated by [Apache Flagon UserALE.js](https://github.com/apache/flagon/tree/master/products/userale/packages/flagon-userale) are sent from the browser to your configured collector endpoint (`url` or `otlpEndpoint`). Because browser security policies enforce CORS on cross-origin `fetch()` requests (especially when `Content-Type: application/json`, `Authorization`, `x-api-key`, or `traceparent` headers are sent), your collector must allow the application's origin and headers.
-For more information and examples on how to enable CORS on your server, please visit [enable-cors.org](http://enable-cors.org/index.html).
+#### OpenTelemetry Collector CORS Configuration (`receivers.otlp.protocols.http`)
+
+When exporting directly from the browser to an OpenTelemetry Collector (`exporterFormat: 'otlp-json'`), enable `cors` on the HTTP receiver in your `otel-collector-config.yaml`:
+
+```yaml
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: 0.0.0.0:4318
+ cors:
+ allowed_origins:
+ - "http://localhost:*"
+ - "https://your-app.example.com"
+ allowed_headers:
+ - "Content-Type"
+ - "Authorization"
+ - "x-api-key"
+ - "traceparent"
+ - "tracestate"
+```
+
+See [`products/userale/example/otel-demo/otel-collector-config.yaml`](https://github.com/apache/flagon/blob/master/products/userale/example/otel-demo/otel-collector-config.yaml) for a complete working example.
+
+---
+
+### Page Unload (`pagehide`) Payload Limits
+
+Browsers enforce a 64 KiB cumulative quota on `fetch(..., { keepalive: true })` requests triggered during `pagehide` (tab close or navigation). UserALE.js's HTTP exporters (`HttpJsonExporter`, `ProtoJsonExporter`, `OtlpJsonExporter`) automatically split large unload queues into $\le 48\text{ KiB}$ chunks via `chunkLogsForKeepalive` so unload flushes preserve custom `Authorization` and `x-api-key` headers without exceeding browser `keepalive` limits.
+
+---
## Contributing
-Contributions are welcome! Simply [submit an issue](https://github.com/apache/flagon-useralejs/issues) for problems
-you encounter or a pull request for your feature or bug fix. The core team will review it and work with you to
-incorporate it into UserALE.js. We also love Pull Requests!
\ No newline at end of file
+Contributions are welcome! See our [Contributing Guide]({{ '/docs/contributing' | prepend: site.baseurl }}) or open an issue on [GitHub](https://github.com/apache/flagon/issues).
\ No newline at end of file