Re: Proposal for Implementation of the Proposed Iceberg REST Spec - Events API

[Adnan Hemani <ad...@snowflake.com.INVALID>]

Hi all,

As per the latest community discussion today (6/17), there was a request to put down the intended use cases for storing Polaris Events in the persistence. There are two main use cases I am intending to support:

[From the Iceberg Events API proposal <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>] Notification of changes / workflow triggers
"The endpoint enables non-catalog systems to efficiently track metadata changes without implementing the full Iceberg REST protocol. Engineers can build event-driven architectures where data quality checks, data processing pipelines, and dashboard refreshes are automatically triggered when relevant events occur. Orchestration tools only need to query the simple /events Endpoint."
[Partially from the Iceberg Events API proposal <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>] Auditing
"Organizations can monitor who modified objects, when events occurred, and what was changed. The [Iceberg Events API] in its current form cannot serve as a replacement to an audit endpoint, as it misses queries, permission events and other events related to the management of the catalog."

While I agree with the above statements regarding auditing capabilities from the Iceberg Events API spec as it is written without Custom Operation support, I believe that we can fit auditing events into this API while still adhering to the spec. However, I am not opposed to a different API endpoint specifically for auditing purposes if the community recommends that. But the PR currently open <https://github.com/apache/polaris/pull/1844> regarding persisting Polaris Events is still required, even if that is the case.

Please let me know if there are any thoughts on this.

Best,
Adnan Hemani



> On May 13, 2025, at 6:35 PM, Adnan Hemani <ad...@snowflake.com> wrote:
> 
> Hi all,
> 
> I am raising a proposal to implement the proposed Iceberg REST specification for the Events API (doc <https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit?pli=1&tab=t.0>, GH <https://github.com/apache/iceberg/pull/12584/files>). It is my understanding that this proposal is close and that we will be required to implement something very close to the current proposal in the near future.
> 
> If Polaris is to implement this API, it will likely need to be through a Persistence instance that the Polaris server can query instantly, as this API will not be asynchronous. Please note, this proposal is not to comment on what events we may emit today or in the future - the scope of this proposal is solely to discuss how we plan to implement the proposed Events API.
> 
> Changes to be made:
> 
> Implement Event storage through the Polaris Persistence layer
> 
> We will store events in a persistence instance of user’s choice - whether they would like the events to be part of the same persistence instance as their Polaris metadata or if they would like for a separate persistence instance. Users will provide the persistence instance by configuring a JDBC string on Polaris startup, similarly to the JDBC string we receive currently from users for the Polaris metadata.
> 
> For concerns regarding scaling of events in the Polaris persistence layer, we can also implement a recommended, optional parameter for an events retention period after which Polaris will asynchronously delete records older than that time period.
> 
> How to Implement Writes to the Polaris Persistence layer
> 
> The way to implement the above proposal would be through implementation of the `PolarisEventListener` <https://github.com/apache/polaris/blob/main/service/common/src/main/java/org/apache/polaris/service/events/PolarisEventListener.java> abstract class. In this implementation, I believe it should not be controversial to state that we cannot block on events to be flushed to persistence due to latency concerns - and as a result, we have two options: 1) a simple in-memory buffer or 2) a file-based buffer. Both buffers would flush after a certain amount of time after the first non-flushed event is written. While option 2 offers a better event durability guarantee in case of disaster recovery, it will come at the cost of additional latency to write to the filesystem. If there are no security concerns regarding writing to the filesystem, I believe this is the recommended way to implement - the additional latency to write to filesystem should not add unreasonable overhead given the right implementation with open filewriters. If writing to the filesystem is not recommended, I’m not sure there is any other way to achieve guaranteed event durability. In both options we can only achieve eventual consistency - to get strong consistency, we will need to implement a way to block the API call until we flush the events to persistence, which I cannot recommend at this time due to latency concerns.
> 
> Please reply to this thread if there are any questions and/or concerns on this proposal. If there are no major concerns within a week, then I will begin implementation.
> 
> Best,
> Adnan Hemani
>