You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "richiebachala (via GitHub)" <gi...@apache.org> on 2026/10/15 03:13:11 UTC
[PR] Document current audit behavior of Polaris events [polaris]
richiebachala opened a new pull request, #5795:
URL: https://github.com/apache/polaris/pull/5795
Adds a **Managing Security > Auditing with Events** page describing what Polaris records today for audit purposes, how events are delivered, and the known limits of that delivery. This is the first step proposed in #5561 and agreed by @adutra there: it documents the status quo and does not take a position on the open architecture questions (durable delivery, fail-closed behavior, events for authentication failures and authorization denials).
The page covers:
- What is recorded: event types and categories, including management operations and credential vending; event metadata; and what `DefaultEventSanitizer` removes.
- What is not recorded: authorization denials (only `BEFORE_*` is emitted; with the built-in authorizer the denial is logged at `INFO`), authentication failures, and external authorizer decisions.
- Listener identifiers, filtering, and an example configuration.
- Limits of the JDBC `events` table: shared datasource, primary key only, no retention or query API, not available with NoSQL persistence.
- Delivery semantics as currently implemented: asynchronous and outside the request transaction, bounded backlog drops, no retry after a listener exception, persistence buffer flush and retry, shutdown versus crash, Kafka and CloudWatch async defaults, and ordering limits.
- Correlating with CloudTrail via STS session tags and session names, and with GCS Data Access logs via principal attribution.
- Investigating a denied request using `X-Request-ID`, the server log, and the event stream.
Each claim was checked against the current `main` code and the generated configuration reference. Nothing here describes in-flight work such as the webhook listener (#5084) or the CloudEvents mapper (#5717).
Related to #5561. This does not fix it: the remaining items in that issue need a dev-list decision.
## Checklist
- [x] ๐ก๏ธ Don't disclose security issues! (contact security@apache.org)
- [x] ๐ Clearly explained why the changes are needed, or linked related issues: Related to #5561
- [x] ๐งช Added/updated tests with good coverage, or manually tested (and explained how): docs only. Built the site with `site/bin/create-static-site.sh`; Hugo reported no errors or warnings. Checked that the page renders, all `ref` anchors (`#events`, `#polarisfeatures`, `#mdc-logging`) resolve, and the Managing Security index links to it. The `rat` audit approves the new file.
- [ ] ๐ก Added comments for complex logic (n/a, docs only)
- [ ] ๐งพ Updated `CHANGELOG.md` (if needed): not added, docs only
- [x] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed)
Drafted with AI assistance. I reviewed each statement against the source before submitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document current audit behavior of Polaris events [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on code in PR #5795:
URL: https://github.com/apache/polaris/pull/5795#discussion_r4428747478
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
Review Comment:
+1 to what @adutra said in multiple comments. I think this page is really about events, not auditing. Polaris events can't be used for auditing as they are today. Auditing needs completeness, but events are incomplete (no denials, no auth failures) and lossy (async and best-effort). Calling the page "Auditing" under Managing Security suggests otherwise.
How about renaming it to "Events" and moving it out of Managing Security? Most of the content is about events in general anyway, and we don't have an events page yet. We can keep the limits section as a heads-up for anyone thinking of using events for auditing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document current audit behavior of Polaris events [polaris]
Posted by "orca-review-bot (via GitHub)" <gi...@apache.org>.
orca-review-bot commented on code in PR #5795:
URL: https://github.com/apache/polaris/pull/5795#discussion_r4341994926
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
+ operation emits its `BEFORE_*` event and no matching `AFTER_*` event. With the
+ built-in authorizer, the server logs the denial at `INFO`, including the
+ principal and the missing privileges.
+* **Authentication failures.** Requests rejected during authentication are not
+ dispatched as events.
+* **Decisions made by external authorizers.** If an
+ [external policy decision point]({{< relref "external-pdp" >}}) is configured,
+ its decision logs are produced by that system, not by Polaris.
+
+HTTP access logs are not a substitute. They do not reliably include the
+authenticated principal.
+
+## Choosing listeners
+
+Enable one or more listeners with `polaris.event-listener.types`. Each listener
+can be limited to specific event types or categories with
+`polaris.event-listener.<name>.enabled-event-types` and
+`polaris.event-listener.<name>.enabled-event-categories`. If neither is set, the
+listener receives all events.
+
+| Listener | Identifier | Destination |
+|----------|------------|-------------|
+| Persistence (in-memory buffer) | `persistence-in-memory-buffer` | `events` table in the Polaris metastore |
+| AWS CloudWatch | `aws-cloudwatch` | CloudWatch Logs |
+| Kafka | `kafka` | A Kafka topic |
+| OpenTelemetry | `opentelemetry` | OpenTelemetry log records |
+
+For example, to send all events to Kafka and to an OpenTelemetry collector:
+
+```properties
+polaris.event-listener.types=kafka,opentelemetry
+polaris.event-listener.kafka.properties.bootstrap.servers=kafka:9092
+polaris.event-listener.kafka.topic=polaris-catalog-events
+quarkus.otel.sdk.disabled=false
+quarkus.otel.logs.enabled=true
+quarkus.otel.exporter.otlp.endpoint=http://otlp-collector:4317
+```
+
+See the [configuration reference]({{% ref "../configuration/configuration-reference#events" %}})
+for all listener settings.
+
+### The `events` table
+
+The persistence listener writes to the `events` table in the same database, and
+through the same datasource, as the catalog metastore. The table has a primary
+key on `event_id` and no other indexes. Polaris provides no retention, purge,
+or query API for it, and users with direct database access can modify it. The
+persistence listener is not available with NoSQL persistence. For an audit trail,
+forward events to an external system instead.
+
+## Delivery guarantees
+
+Event delivery is best-effort. Operators should plan for the following:
+
+* **Events are delivered asynchronously.** They are not written in the same
+ transaction as the operation they describe, and a delivery failure does not
+ fail the request.
+* **A bounded backlog drops events.** If
+ `polaris.event-listener.listener-backlog.queue-size` is set, events are dropped
+ and logged at `ERROR` when a listener's backlog is full. The default is unbounded.
+* **A failing listener loses the event.** If a listener throws while handling an
+ event, the error is logged and the event is not retried. Other listeners are
+ not affected.
+* **The persistence listener buffers in memory.** Events are flushed after
+ `polaris.event-listener.persistence-in-memory-buffer.buffer-time` (default
+ `5000ms`) or after `max-buffer-size` events (default `5`). A failed flush is
+ retried, then the batch is logged as failed and discarded. On shutdown,
+ Polaris attempts to flush buffered events and discards events that arrive
+ after shutdown begins. Buffered events are lost if the process terminates
+ abruptly.
+* **Kafka and CloudWatch send asynchronously by default.** Send failures are
+ logged and not retried by Polaris. Setting `synchronous-mode=true` makes the
+ listener wait for each send. It does not make the originating request wait or
+ fail.
+* **Ordering is limited.** There is no ordering guarantee across listeners. The
+ Kafka listener uses the event ID as the record key, so the Kafka listener
+ configuration recommends a single-partition topic to preserve FIFO order.
+
+## Correlating with storage access logs
+
+Catalog events do not record reads and writes to table data. When Polaris vends
+storage credentials, the cloud provider's access logs record the data access.
+
+* **AWS:** Polaris can add session tags to the STS `AssumeRole` request used for
+ credential vending. The tags appear in CloudTrail and can be joined with
+ Polaris events. Session tags are disabled by default. Enable them with the
+ `SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL` feature flag, which requires the IAM
+ role trust policy to allow `sts:TagSession`. The
+ `SESSION_NAME_FIELDS_IN_SUBSCOPED_CREDENTIAL` flag adds context to the role
+ session name instead.
+* **Google Cloud Storage:** with `GCS_PRINCIPAL_ATTRIBUTION_ENABLED`, the Polaris
+ principal appears in Cloud Storage Data Access logs. It requires additional
+ Workload Identity Federation settings.
+
+See the [feature flags reference]({{% ref "../configuration/configuration-reference#polarisfeatures" %}})
+for the options and requirements of each flag.
+
+## Investigating a denied request
+
+Polaris returns the request ID in the `X-Request-ID` response header. A client
+can also set the header on the request. To investigate a denial:
+
+1. Get the request ID from the client's response or logs.
+2. Search the Polaris server log for that request ID. With the built-in
+ authorizer, the log line for the denial names the principal, the operation,
+ and the missing privileges. With an external authorizer, check that system's
+ decision logs.
+3. In the event sink, find events with the same request ID. A denied operation
+ shows a `BEFORE_*` event without a matching `AFTER_*` event.
Review Comment:
A `BEFORE_*` with no `AFTER_*` isn't specific to denials. Any failed request looks the same in the event stream: a 404, a commit conflict, a validation error. So step 3 on its own can't tell you it was a denial. Can we say that here, and point back to the server log line from step 2 as what actually confirms it?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
Review Comment:
Listeners that implement `RawEventAccess` skip the sanitizer entirely (`PolarisEventListeners.deliverEvent`). They get the full catalog and principal objects. Since this paragraph puts the responsibility on custom-listener authors, can we mention that opt-out here so operators know to check custom listeners for it?
<sub>๐ค AI-generated review comment โ agents make mistakes, so please verify before acting on it.</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document Polaris events and their limits for auditing [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain merged PR #5795:
URL: https://github.com/apache/polaris/pull/5795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document Polaris events and their limits for auditing [polaris]
Posted by "flyrain (via GitHub) via issues" <is...@polaris.apache.org>.
flyrain commented on PR #5795:
URL: https://github.com/apache/polaris/pull/5795#issuecomment-6582918751
Thanks @richiebachala for the PR. Thanks everyone for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document Polaris events and their limits for auditing [polaris]
Posted by "richiebachala (via GitHub) via issues" <is...@polaris.apache.org>.
richiebachala commented on code in PR #5795:
URL: https://github.com/apache/polaris/pull/5795#discussion_r4430792342
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
Review Comment:
Agreed, thanks @flyrain. Renamed the page to "Events" and moved it to the top level next to Telemetry, out of Managing Security. The intro now describes events in general, and the warning at the top states up front that events are not a complete audit trail (no denials or auth failures, asynchronous and best-effort), linking to the coverage and delivery sections. Content is otherwise unchanged. @adutra @nandorKollar @dimas-b, sorry for the churn after your approvals; could you take another quick look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document current audit behavior of Polaris events [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #5795:
URL: https://github.com/apache/polaris/pull/5795#discussion_r4340465184
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
Review Comment:
Some portions of this doc read more like a generic doc page about events (which we don't have yet today, and that's unfortunate), rather than a security-focused anaylizis of Polaris events.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
+ operation emits its `BEFORE_*` event and no matching `AFTER_*` event. With the
+ built-in authorizer, the server logs the denial at `INFO`, including the
+ principal and the missing privileges.
+* **Authentication failures.** Requests rejected during authentication are not
+ dispatched as events.
Review Comment:
Emitting events for failed authentications would open a possibility for DoS attacks, imo.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
Review Comment:
This is the default sanitizer, but it can be changed in a downstream build of Polaris, FYI.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
+ operation emits its `BEFORE_*` event and no matching `AFTER_*` event. With the
+ built-in authorizer, the server logs the denial at `INFO`, including the
+ principal and the missing privileges.
+* **Authentication failures.** Requests rejected during authentication are not
+ dispatched as events.
+* **Decisions made by external authorizers.** If an
+ [external policy decision point]({{< relref "external-pdp" >}}) is configured,
+ its decision logs are produced by that system, not by Polaris.
+
+HTTP access logs are not a substitute. They do not reliably include the
+authenticated principal.
+
+## Choosing listeners
+
+Enable one or more listeners with `polaris.event-listener.types`. Each listener
+can be limited to specific event types or categories with
+`polaris.event-listener.<name>.enabled-event-types` and
+`polaris.event-listener.<name>.enabled-event-categories`. If neither is set, the
+listener receives all events.
+
+| Listener | Identifier | Destination |
+|----------|------------|-------------|
+| Persistence (in-memory buffer) | `persistence-in-memory-buffer` | `events` table in the Polaris metastore |
+| AWS CloudWatch | `aws-cloudwatch` | CloudWatch Logs |
+| Kafka | `kafka` | A Kafka topic |
+| OpenTelemetry | `opentelemetry` | OpenTelemetry log records |
+
+For example, to send all events to Kafka and to an OpenTelemetry collector:
+
+```properties
+polaris.event-listener.types=kafka,opentelemetry
+polaris.event-listener.kafka.properties.bootstrap.servers=kafka:9092
+polaris.event-listener.kafka.topic=polaris-catalog-events
+quarkus.otel.sdk.disabled=false
+quarkus.otel.logs.enabled=true
+quarkus.otel.exporter.otlp.endpoint=http://otlp-collector:4317
+```
+
+See the [configuration reference]({{% ref "../configuration/configuration-reference#events" %}})
+for all listener settings.
+
+### The `events` table
+
+The persistence listener writes to the `events` table in the same database, and
+through the same datasource, as the catalog metastore. The table has a primary
+key on `event_id` and no other indexes. Polaris provides no retention, purge,
+or query API for it, and users with direct database access can modify it. The
+persistence listener is not available with NoSQL persistence. For an audit trail,
Review Comment:
That is also unfortunate.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
+ operation emits its `BEFORE_*` event and no matching `AFTER_*` event. With the
+ built-in authorizer, the server logs the denial at `INFO`, including the
+ principal and the missing privileges.
+* **Authentication failures.** Requests rejected during authentication are not
+ dispatched as events.
+* **Decisions made by external authorizers.** If an
+ [external policy decision point]({{< relref "external-pdp" >}}) is configured,
+ its decision logs are produced by that system, not by Polaris.
+
+HTTP access logs are not a substitute. They do not reliably include the
+authenticated principal.
+
+## Choosing listeners
+
+Enable one or more listeners with `polaris.event-listener.types`. Each listener
+can be limited to specific event types or categories with
+`polaris.event-listener.<name>.enabled-event-types` and
+`polaris.event-listener.<name>.enabled-event-categories`. If neither is set, the
+listener receives all events.
+
+| Listener | Identifier | Destination |
+|----------|------------|-------------|
+| Persistence (in-memory buffer) | `persistence-in-memory-buffer` | `events` table in the Polaris metastore |
+| AWS CloudWatch | `aws-cloudwatch` | CloudWatch Logs |
+| Kafka | `kafka` | A Kafka topic |
+| OpenTelemetry | `opentelemetry` | OpenTelemetry log records |
+
+For example, to send all events to Kafka and to an OpenTelemetry collector:
+
+```properties
+polaris.event-listener.types=kafka,opentelemetry
+polaris.event-listener.kafka.properties.bootstrap.servers=kafka:9092
+polaris.event-listener.kafka.topic=polaris-catalog-events
+quarkus.otel.sdk.disabled=false
+quarkus.otel.logs.enabled=true
+quarkus.otel.exporter.otlp.endpoint=http://otlp-collector:4317
+```
+
+See the [configuration reference]({{% ref "../configuration/configuration-reference#events" %}})
+for all listener settings.
+
+### The `events` table
+
+The persistence listener writes to the `events` table in the same database, and
+through the same datasource, as the catalog metastore. The table has a primary
+key on `event_id` and no other indexes. Polaris provides no retention, purge,
+or query API for it, and users with direct database access can modify it. The
+persistence listener is not available with NoSQL persistence. For an audit trail,
+forward events to an external system instead.
+
+## Delivery guarantees
+
+Event delivery is best-effort. Operators should plan for the following:
+
+* **Events are delivered asynchronously.** They are not written in the same
+ transaction as the operation they describe, and a delivery failure does not
+ fail the request.
+* **A bounded backlog drops events.** If
+ `polaris.event-listener.listener-backlog.queue-size` is set, events are dropped
+ and logged at `ERROR` when a listener's backlog is full. The default is unbounded.
+* **A failing listener loses the event.** If a listener throws while handling an
+ event, the error is logged and the event is not retried. Other listeners are
+ not affected.
+* **The persistence listener buffers in memory.** Events are flushed after
+ `polaris.event-listener.persistence-in-memory-buffer.buffer-time` (default
+ `5000ms`) or after `max-buffer-size` events (default `5`). A failed flush is
+ retried, then the batch is logged as failed and discarded. On shutdown,
+ Polaris attempts to flush buffered events and discards events that arrive
+ after shutdown begins. Buffered events are lost if the process terminates
+ abruptly.
+* **Kafka and CloudWatch send asynchronously by default.** Send failures are
+ logged and not retried by Polaris. Setting `synchronous-mode=true` makes the
+ listener wait for each send. It does not make the originating request wait or
+ fail.
+* **Ordering is limited.** There is no ordering guarantee across listeners. The
+ Kafka listener uses the event ID as the record key, so the Kafka listener
+ configuration recommends a single-partition topic to preserve FIFO order.
+
+## Correlating with storage access logs
+
+Catalog events do not record reads and writes to table data. When Polaris vends
+storage credentials, the cloud provider's access logs record the data access.
+
+* **AWS:** Polaris can add session tags to the STS `AssumeRole` request used for
+ credential vending. The tags appear in CloudTrail and can be joined with
+ Polaris events. Session tags are disabled by default. Enable them with the
+ `SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL` feature flag, which requires the IAM
+ role trust policy to allow `sts:TagSession`. The
+ `SESSION_NAME_FIELDS_IN_SUBSCOPED_CREDENTIAL` flag adds context to the role
+ session name instead.
+* **Google Cloud Storage:** with `GCS_PRINCIPAL_ATTRIBUTION_ENABLED`, the Polaris
+ principal appears in Cloud Storage Data Access logs. It requires additional
+ Workload Identity Federation settings.
+
+See the [feature flags reference]({{% ref "../configuration/configuration-reference#polarisfeatures" %}})
+for the options and requirements of each flag.
+
+## Investigating a denied request
+
+Polaris returns the request ID in the `X-Request-ID` response header. A client
Review Comment:
The header name is modifiable, iirc.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
Review Comment:
I advocated a long time ago for failure events. This now brings a good argument in favor of introducing that.
##########
site/content/in-dev/unreleased/managing-security/auditing.md:
##########
@@ -0,0 +1,183 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: Auditing with Events
+type: docs
+weight: 400
+---
+
+Polaris emits structured events for catalog and management operations. Operators
+can forward these events to an external system to build an audit trail. This page
+describes what Polaris records today, how events are delivered, and the known
+limits of that delivery. It documents current behavior and does not prescribe a
+compliance architecture.
+
+{{< alert warning >}}
+Polaris is not a security information and event management (SIEM) system. Retention,
+access control, tamper evidence, and querying of audit records are properties of
+the system that receives the events, not of Polaris.
+{{< /alert >}}
+
+## What is recorded
+
+Each event has a `PolarisEventType`, for example `BEFORE_LOAD_TABLE` and
+`AFTER_LOAD_TABLE`. Most operations emit a `BEFORE_*` event when the request
+starts and an `AFTER_*` event when it completes successfully. Event types are
+grouped into categories such as `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW`,
+`PRINCIPAL`, `PRINCIPAL_ROLE`, `CATALOG_ROLE`, and `CREDENTIAL`.
+
+Events cover both catalog operations and management operations. For example:
+
+* Loading a table: `BEFORE_LOAD_TABLE` and `AFTER_LOAD_TABLE`.
+* Vending storage credentials: `BEFORE_LOAD_CREDENTIALS` and `AFTER_LOAD_CREDENTIALS`.
+* Creating a catalog or principal: `BEFORE_CREATE_CATALOG`, `BEFORE_CREATE_PRINCIPAL`,
+ and their `AFTER_*` counterparts.
+* Rotating principal credentials: `BEFORE_ROTATE_CREDENTIALS` and `AFTER_ROTATE_CREDENTIALS`.
+* Granting and revoking privileges and roles, for example
+ `BEFORE_ADD_GRANT_TO_CATALOG_ROLE` and `BEFORE_ASSIGN_PRINCIPAL_ROLE`.
+
+Every event carries metadata: an event ID, a timestamp, and the realm. When
+available, it also carries the authenticated principal, the request ID, and the
+OpenTelemetry trace context. The principal is absent for events dispatched
+before authentication completes.
+
+Events pass through `DefaultEventSanitizer` before reaching listeners. By default,
+it removes full catalog and principal objects, principal update requests, and
+notification requests. When it removes a catalog object, it keeps the catalog
+name. Additional attributes can be removed with
+`polaris.event-listener.denylisted-attributes`. Each listener then writes only
+selected fields. Custom listeners are responsible for not forwarding sensitive
+attributes.
+
+### What is not recorded
+
+* **Authorization denials.** There is no event type for a denial. A denied
+ operation emits its `BEFORE_*` event and no matching `AFTER_*` event. With the
+ built-in authorizer, the server logs the denial at `INFO`, including the
+ principal and the missing privileges.
+* **Authentication failures.** Requests rejected during authentication are not
+ dispatched as events.
+* **Decisions made by external authorizers.** If an
+ [external policy decision point]({{< relref "external-pdp" >}}) is configured,
+ its decision logs are produced by that system, not by Polaris.
+
+HTTP access logs are not a substitute. They do not reliably include the
+authenticated principal.
+
+## Choosing listeners
+
+Enable one or more listeners with `polaris.event-listener.types`. Each listener
+can be limited to specific event types or categories with
+`polaris.event-listener.<name>.enabled-event-types` and
+`polaris.event-listener.<name>.enabled-event-categories`. If neither is set, the
+listener receives all events.
+
+| Listener | Identifier | Destination |
+|----------|------------|-------------|
+| Persistence (in-memory buffer) | `persistence-in-memory-buffer` | `events` table in the Polaris metastore |
+| AWS CloudWatch | `aws-cloudwatch` | CloudWatch Logs |
+| Kafka | `kafka` | A Kafka topic |
+| OpenTelemetry | `opentelemetry` | OpenTelemetry log records |
+
+For example, to send all events to Kafka and to an OpenTelemetry collector:
+
+```properties
+polaris.event-listener.types=kafka,opentelemetry
+polaris.event-listener.kafka.properties.bootstrap.servers=kafka:9092
+polaris.event-listener.kafka.topic=polaris-catalog-events
+quarkus.otel.sdk.disabled=false
+quarkus.otel.logs.enabled=true
+quarkus.otel.exporter.otlp.endpoint=http://otlp-collector:4317
+```
+
+See the [configuration reference]({{% ref "../configuration/configuration-reference#events" %}})
+for all listener settings.
+
+### The `events` table
+
+The persistence listener writes to the `events` table in the same database, and
+through the same datasource, as the catalog metastore. The table has a primary
+key on `event_id` and no other indexes. Polaris provides no retention, purge,
+or query API for it, and users with direct database access can modify it. The
+persistence listener is not available with NoSQL persistence. For an audit trail,
+forward events to an external system instead.
+
+## Delivery guarantees
+
+Event delivery is best-effort. Operators should plan for the following:
+
+* **Events are delivered asynchronously.** They are not written in the same
+ transaction as the operation they describe, and a delivery failure does not
+ fail the request.
+* **A bounded backlog drops events.** If
+ `polaris.event-listener.listener-backlog.queue-size` is set, events are dropped
+ and logged at `ERROR` when a listener's backlog is full. The default is unbounded.
+* **A failing listener loses the event.** If a listener throws while handling an
+ event, the error is logged and the event is not retried. Other listeners are
+ not affected.
+* **The persistence listener buffers in memory.** Events are flushed after
+ `polaris.event-listener.persistence-in-memory-buffer.buffer-time` (default
+ `5000ms`) or after `max-buffer-size` events (default `5`). A failed flush is
+ retried, then the batch is logged as failed and discarded. On shutdown,
+ Polaris attempts to flush buffered events and discards events that arrive
+ after shutdown begins. Buffered events are lost if the process terminates
+ abruptly.
+* **Kafka and CloudWatch send asynchronously by default.** Send failures are
+ logged and not retried by Polaris. Setting `synchronous-mode=true` makes the
+ listener wait for each send. It does not make the originating request wait or
+ fail.
+* **Ordering is limited.** There is no ordering guarantee across listeners. The
+ Kafka listener uses the event ID as the record key, so the Kafka listener
+ configuration recommends a single-partition topic to preserve FIFO order.
+
+## Correlating with storage access logs
+
+Catalog events do not record reads and writes to table data. When Polaris vends
+storage credentials, the cloud provider's access logs record the data access.
+
+* **AWS:** Polaris can add session tags to the STS `AssumeRole` request used for
+ credential vending. The tags appear in CloudTrail and can be joined with
+ Polaris events. Session tags are disabled by default. Enable them with the
+ `SESSION_TAGS_IN_SUBSCOPED_CREDENTIAL` feature flag, which requires the IAM
+ role trust policy to allow `sts:TagSession`. The
+ `SESSION_NAME_FIELDS_IN_SUBSCOPED_CREDENTIAL` flag adds context to the role
+ session name instead.
+* **Google Cloud Storage:** with `GCS_PRINCIPAL_ATTRIBUTION_ENABLED`, the Polaris
+ principal appears in Cloud Storage Data Access logs. It requires additional
+ Workload Identity Federation settings.
+
+See the [feature flags reference]({{% ref "../configuration/configuration-reference#polarisfeatures" %}})
+for the options and requirements of each flag.
+
+## Investigating a denied request
Review Comment:
I would rename this section "Correlating with request IDs" or something like that. As request IDs can be used to correlate any request, not just denied ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document current audit behavior of Polaris events [polaris]
Posted by "richiebachala (via GitHub)" <gi...@apache.org>.
richiebachala commented on PR #5795:
URL: https://github.com/apache/polaris/pull/5795#issuecomment-6390303140
Thanks for the review @adutra! Pushed an update:
- Reframed the intro around audit questions (what's attributable, what's missing, where records can be lost) and dropped the generic listener config example in favor of the config reference. Happy to trim further if specific sections still read too generic.
- Noted that `DefaultEventSanitizer` is only the default: a custom build can replace the `EventSanitizer` bean, and listeners implementing `RawEventAccess` bypass sanitization (also raised by the bot review).
- Clarified that `X-Request-ID` is the default header name and is configurable.
- Renamed the last section to "Correlating with request IDs", with the denial walkthrough kept as an example. Step 3 now notes that `BEFORE_*` without `AFTER_*` appears for any failed request, so the server log confirms a denial.
On failure events and the DoS concern for failed authentication: agreed that both deserve discussion; I'll pick them up in #5561 / the dev list since this PR only documents current behavior.
Could you also approve the pending `CI/PR` and `Security` workflow runs? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Document current audit behavior of Polaris events [polaris]
Posted by "richiebachala (via GitHub)" <gi...@apache.org>.
richiebachala commented on PR #5795:
URL: https://github.com/apache/polaris/pull/5795#issuecomment-6184007314
@adutra this is the docs-only first step you agreed to on #5561. Could you approve the workflow runs when you have a moment? The `CI/PR` and `Security` runs are waiting for maintainer approval. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org