You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "donalmag (via GitHub)" <gi...@apache.org> on 2025/07/28 12:37:54 UTC
[PR] HDDS-13513 Ozone Event Notification Design [ozone]
donalmag opened a new pull request, #8871:
URL: https://github.com/apache/ozone/pull/8871
## What changes were proposed in this pull request?
Provide a one-liner summary of the changes in the PR **Title** field above.
It should be in the form of `HDDS-1234. Short summary of the change`.
Please describe your PR in detail:
* What changes are proposed in the PR? and Why? It would be better if it is written from third person's
perspective not just for the reviewer.
* Provide as much context and rationale for the pull request as possible. It could be copy-paste from
the Jira's description if the jira is well defined.
* If it is complex code, describe the approach used to solve the issue. If possible attach design doc,
issue investigation, github discussion, etc.
Examples of well-written pull requests:
* [#3980](https://github.com/apache/ozone/pull/3980)
* [#5265](https://github.com/apache/ozone/pull/5265)
* [#4701](https://github.com/apache/ozone/pull/4701)
* [#5283](https://github.com/apache/ozone/pull/5283)
* [#5300](https://github.com/apache/ozone/pull/5300)
## What is the link to the Apache JIRA
Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
(Please replace this section with the link to the Apache JIRA)
## How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501393336
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
(Just a passing comment) In the future, we can also support CloudEvents (https://cloudevents.io/) which is also supported by major cloud provider (AWS EventBridge, GCP Eventarc, Azure Event Grid) and Apache project like EventMesh (https://eventmesh.apache.org/docs/design-document/event-handling-and-integration/cloudevents/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501117640
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
Initial plan was for retention policy to be based on entry count to keep the table in a bounded size. It looks like we missed writing this down though. @donalmag can you add this detail to the doc?
The OM has no knowledge of what the plugins are doing with the data they read, so the plugins must be able to process the events at a rate at least equal to the OM writing them otherwise they will miss updates. This includes failed processing due to one of the plugin's targets being down. This prevents things from backing up in the system and gives a clear SLA on event consumption, after which events may be missed.
Plugins have a large degree of flexibility and could choose to buffer events locally if the target is down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3629574903
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2426736831
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
Review Comment:
Since we are persisting the ledger in RocksDB, it will automatically be transmitted as part of the Ratis snapshot install process, which sends over the entire RocksDB of the leader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501393336
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
(Just a pass-through comment) In the future, we can also support CloudEvents (https://cloudevents.io/) which is also supported by major cloud provider (AWS EventBridge, GCP Eventarc, Azure Event Grid) and Apache project like (https://eventmesh.apache.org/docs/design-document/event-handling-and-integration/cloudevents/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501632028
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
Will the Plugins run inside OM, or run as a independent process? If a independent process, Plugins can do what ever they want. If they are running inside OM, as part of OM, they should use resources with restraint, for example memory and disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3527519300
> > use this Feature Branch https://github.com/apache/ozone/tree/HDDS-13513_Event_Notification_FeatureBranch
>
> @swamirishi if you push and then delete a [feature branch](https://github.com/apache/ozone/activity?ref=HDDS-13513_Event_Notification_Design), please make sure to cancel the corresponding [workflow run](https://github.com/apache/ozone/actions/runs/19330005767).
But not the run for the feature branch you intend to keep.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2523136030
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
This should not be that big of an issue if we use the deleteRange api which would add a single tombstone.
https://github.com/apache/ozone/pull/8871/files#r2510854640
This table should not be that big in general I believe the table would have at the most 50-100 million entries which is not a lot. Given that the Event object is going to be very lightweight.
This coupled along with the deleteRange we should be ok even in terms of the table iterator. We wouldn't even need the BackgroundCompactionService to kick in here
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
This should not be that big of an issue if we use the deleteRange api which would add a single range tombstone.
https://github.com/apache/ozone/pull/8871/files#r2510854640
This table should not be that big in general I believe the table would have at the most 50-100 million entries which is not a lot. Given that the Event object is going to be very lightweight.
This coupled along with the deleteRange we should be ok even in terms of the table iterator. We wouldn't even need the BackgroundCompactionService to kick in here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-4703591115
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3156702974
Sharing recordings of design discussions that happened in the last two week's community sync meetings:
https://cloudera.zoom.us/rec/share/TwWtWixjJ3tF8wM5n7o7Gxx3CEeMv32uJASsvqk-UfuoBJIbaU2Z2MIqAELykPI5.5JXR4fzks37Lsi1n?startTime=1753718482000
Passcode: d1NL..z7
https://cloudera.zoom.us/rec/share/Sc1klNNvDvIZN0n5yZVxOrV2jqdgT-djeRFke8y6UCPGUnNmzJFV2VMqbD4vcEEb.0myqZ0KY3JuiNXdn?startTime=1754323258000
Passcode: 8j.6reXC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2237286704
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
+
+# Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce a new OzoneManager node in the ratis ring in LISTENER mode.
+The node will be defined up-front in ozone-site.xml.
+This node will maintain metadata similar to the other OM nodes however it will generate notifications for events after they have been successfully commited.
+This node will not be able to become a leader, so the notification process will not impact client requests.
+
+
+
+#### Component
+
+Implementation of this feature requires changes to the OzoneManager:
+ - Implement an agnostic notification interface
+ - Add support for a LISTENER node
+ - Add a hook in the OMRequest exection flow to generate notifications
+
+#### Component
+
+Introduces a new field on OMAdminProtocol.proto to identify the notify node:
+
+```protobuf
+message OMNodeInfo {
+required string nodeID = 1;
+required string hostAddress = 2;
+required uint32 rpcPort = 3;
+required uint32 ratisPort = 4;
+optional NodeState nodeState = 5 [default=ACTIVE];
+optional bool isNotifyNode = 6 [default=false];
+}
+```
+
+## Performance
+
+While this is a synchronous only approach any latency between notification target and OM should not impact the performance of client requests as the notification does not run on the leader.
+
+## Configuration
+
+This is the configuration to enable event notification on node on1.
+
+```xml
+<property>
+ <name>ozone.event.notification.node</name>
+ <value>on1</value>
+</property>
+```
+Example configuration to provide a kafka instance:
+```xml
+
+<configration>
Review Comment:
Optionally, support AWS's PutBucketNotificationConfiguration and GetBucketNotificationConfiguration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2509554503
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
Okay, if the first version is plugin will run in inside OM, please add a clarify that plugin should use resources shared by OM with restraint, especially memory resources.
If the plugin is not configured, "event data capture" should not start, right? Otherwise, the resource is spent but not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8871: HDDS-13513 Ozone Event Notification Design
URL: https://github.com/apache/ozone/pull/8871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r3276165022
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
I have two thoughts:
1. we should support dynamic configuration(throught s3 endpoint) of destinations and only store the `CompletedRequestInfo` if there is destination existing(event notifications are not retroactive). and looks like we could support more complex configurtion, for each event notification:
- Event name,
- Prefix (optional)
- Suffix (optional)
- Destination (only one for one event notification)
ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html#:~:text=In%20the%20General,and%20Lambda.
so you might want to create a separate table to store these.
3. For the "retention policy" and "Offset tracking and persistence",
I think we should introduce a background service to mange the watermark of all downstream sink(eg. kafka destination).
the background service will basically loop on leader periodically:
1. check if we have been suspended and `isLeaderReady()`
1. `restart_txnid` = query smallest txn id in complete info with `iter.Seek();`
2. `confirmed_txnid` = min(txn id that all downstream have committed write/persist)
4. check the retention policy, assume we limit `max_record_keep_size` configured.
1. `reclaim_txnid = iter.key()`, after `iter.SeekToLast()` and run `iter.Prev()` for `max_record_keep_size` times
2. then `reclaim_range = [restart_txnid, max(reclaim_txnid, confirmed_txnid)]`
5. submit a raft request/response to delete `reclaim_range` with rocksdb `deleteRange`.
6. sleep a while and goto first step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2246298742
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
Review Comment:
Seems like this field would be redundant given the event name. Same with `CreateDirectoryRequest`.
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
+
+# Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce a new OzoneManager node in the ratis ring in LISTENER mode.
+The node will be defined up-front in ozone-site.xml.
+This node will maintain metadata similar to the other OM nodes however it will generate notifications for events after they have been successfully commited.
+This node will not be able to become a leader, so the notification process will not impact client requests.
+
+
+
+#### Component
+
+Implementation of this feature requires changes to the OzoneManager:
+ - Implement an agnostic notification interface
+ - Add support for a LISTENER node
+ - Add a hook in the OMRequest exection flow to generate notifications
+
+#### Component
+
+Introduces a new field on OMAdminProtocol.proto to identify the notify node:
+
+```protobuf
+message OMNodeInfo {
+required string nodeID = 1;
+required string hostAddress = 2;
+required uint32 rpcPort = 3;
+required uint32 ratisPort = 4;
+optional NodeState nodeState = 5 [default=ACTIVE];
+optional bool isNotifyNode = 6 [default=false];
+}
+```
+
+## Performance
+
+While this is a synchronous only approach any latency between notification target and OM should not impact the performance of client requests as the notification does not run on the leader.
Review Comment:
I'm not sure "synchronous" is the best term to use here. [Ceph's definitions](https://docs.ceph.com/en/quincy/radosgw/notifications/#synchronous-notifications) are that synchronous blocks the writer and asynchronous does not. It's probably clearer to use similar definitions here, so the system would be asynchronous. I think the point being made here is that the notifier will block until the consumer acks the event but that this will not block the writer.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
Review Comment:
Yes this is probably the best way to do it with rename or delete directory generating a single event. These are atomic operations on the Ozone cluster, so ideally the consumer would see them that way as well.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
+
+Therefore to produce notifications using the standard S3 event
+notification semantics for FSO directory renames we would need to
+consider the trade-offs between compatibility with the normal S3
+semantics for renames vs a custom event type for directory renames.
+
+### most compatible approach
+
+We could introduce some additional processing before emitting notification
+events in the case of a directory rename which "gathers together" (prior
+to the change being committed to the DB) the child objects impacted by
+the directory rename and emits pairs of delete/create events for each
+key (as described above)
+
+Pros:
+- standard S3 event notification rename semantics
+
+Cons:
+- additional processing to pull together the events. This could mean an
+ unknown amount of additional processing for large directory renames.
+- could be a performance drag if performed on the leader
+
+### custom event type
+
+Conversely - we could opt to not try to be fully compliant with existing S3 event notification
+semantics since the schema was designed for non-hierarchical filesystems and
+instead create some custom event extension (e.g. ObjectRenamed:) and
+emit just a single event for directory renames which specifies only the parent
+paths impacted by the rename:
+
+e.g.
+```
+ eventName=ObjectReanmed:Reanmed, fromKey=myfiles, toKey=myfiles-RENAMED
+```
+
+.. it would then be up to the notification consumer to deal with the
+different rename event semantics (i.e. that only the parent names were
+notified and not the impacted child objects).
+
+This is the same semantics used in the HDFS inotify directory rename
+event (see below).
+
+Pros:
+- no additional processing when emitting events
+
+Cons:
+- non-standard S3 event notification semantics
+
+NOTE: directory rename is just one example of a hierarchical FSO
+operation which impacts child objects. There may be other Ozone
+hierarchical FSO operations which will need be catered for in a similar
+way (recursive delete?)
Review Comment:
I think directory delete and rename are the only two that fit this category. We do have atomic recursive directory delete.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
Review Comment:
We don't support atime ([only mtime](https://github.com/apache/ozone/blob/5d1b43d44fa435b5a304bd1d99e8fa4a60d092cf/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeySetTimesRequest.java#L164)) because atime turns all read operations into write operations which kills performance.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
+
+Therefore to produce notifications using the standard S3 event
+notification semantics for FSO directory renames we would need to
+consider the trade-offs between compatibility with the normal S3
+semantics for renames vs a custom event type for directory renames.
+
+### most compatible approach
+
+We could introduce some additional processing before emitting notification
+events in the case of a directory rename which "gathers together" (prior
+to the change being committed to the DB) the child objects impacted by
+the directory rename and emits pairs of delete/create events for each
+key (as described above)
+
+Pros:
+- standard S3 event notification rename semantics
+
+Cons:
+- additional processing to pull together the events. This could mean an
+ unknown amount of additional processing for large directory renames.
+- could be a performance drag if performed on the leader
+
+### custom event type
+
+Conversely - we could opt to not try to be fully compliant with existing S3 event notification
+semantics since the schema was designed for non-hierarchical filesystems and
+instead create some custom event extension (e.g. ObjectRenamed:) and
+emit just a single event for directory renames which specifies only the parent
+paths impacted by the rename:
+
+e.g.
+```
+ eventName=ObjectReanmed:Reanmed, fromKey=myfiles, toKey=myfiles-RENAMED
+```
+
+.. it would then be up to the notification consumer to deal with the
+different rename event semantics (i.e. that only the parent names were
+notified and not the impacted child objects).
+
+This is the same semantics used in the HDFS inotify directory rename
+event (see below).
+
+Pros:
+- no additional processing when emitting events
+
+Cons:
+- non-standard S3 event notification semantics
Review Comment:
I think this is acceptable for FSO buckets. FSO buckets accessed through S3 API do compatibility on a best-effort basis, but there are some things that just won't work. For example writing an object called `/bucket/prefix` then writing another object called `/bucket/prefix/key` is valid in S3 and OBS but not FSO (the second object would try to create `prefix` as a directory). IMO it is ok for S3 event notifications to make similar tradeoffs when used with FSO buckets.
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
Review Comment:
Interesting that [S3 does not guarantee event order](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#event-ordering-and-duplicate-events). It seems like this would make it difficult to build any sort of replication based system around these events. We may be able to support this depending how we are working with the Ratis logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2498289836
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
If the connection to external target fails, will the events in DB still be removed by the retention policy? What's the rule of retention policy will follow, entry count based, time window based, or other algorithms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2504351868
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
Current plan is plugins will run inside OM. The overhead of managing extra processes for each event handler would be difficult for admins. This includes each separate process needing its own config file, logs, and jmx server/endpoint. CPU and disk concerns exist for either option because the separate process would still need to be co-located on the OM to read from RocksDB, so it would be sharing CPU and disk bandwidth with the OM either way.
In practice the event plugins should be lightweight as they are just doing incremental RocksDB iterations, some translation, and a push. We also have control over which plugins are supported in the main Ozone repo, and can place guidelines on how many plugins are used simultaneously. We can choose to maintain simple/common ones while anything more complex a user would have to build and maintain themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3267172948
Discussed in the community sync today. We agreed approach 2 for proto schema looks better since it allows us to choose which fields to expose to consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3700890735
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3177110525
Here is the recording of today's discussion:
https://cloudera.zoom.us/rec/share/gV55URMxUdXpdq8UYHy4EtyoZ3a1Vro1lJYUWeQDA5smHHuVLyr4K7zqY5ddgbIK.h5bSNPYJvhDEbwaV
Passcode: jj6oSj4?
Here is a high level summary of what was discussed. Feel free to add other details I may have missed.
---
The high level design discussed involves the Ozone Manager persisting a bounded number of events to RocksDB as they occur. Plugins decoupled from the OM can consume these events by iterating the RocksDB column family which stores these events. Plugins would have freedom to make many decisions as needed for their use case:
- How they want to persist the index of the event they last left off on
- Which node should have the currently active plugin
- Whether the plugin pushes to a single source or multiple sources
- Whether event delivery to the source is blocking or non-blocking
- What schema they want to output
Internally, the Ozone Manager can persist events to its state machine as part of the `applyTransaction` execution of relevant requests. This ensures high availability of events for at least once delivery. Events can be written to a new column family in RocksDB using a key based on an incrementing transaction ID. The OM would have a configuration like `ozone.om.events.max` which would define the maximum number of events that would remain persisted. Once this number is exceeded, older events may be deleted by the OM with a RocksDB range delete.
This provides a spec for plugins consuming events: They can be guaranteed to see at least the last `ozone.om.max.events`, and they can use the transaction ID of the last event they processed to resume processing newer events.
To run plugins, a few options were discussed:
1. Run each plugin as a thread within the OM
- **Pros**:
- Easiest process and configuration management
- Easy for plugins to check if the current OM is the leader, and use this as their run condition
- Plugins can add metrics to the OM's JMX endpoint for monitoring.
- **Cons**:
- Potentially more load on the OM if there are a lot of plugins pushing information or churning iterator objects.
- Adding or removing plugins would likely require (rolling) restart of OMs since the classpath must be updated
- If a plugin's thread crashes unexpectedly, either the OM must be restarted to revive it, or a management CLI to start or stop plugins must be added.
2. Run each plugin as a process on each OM using a read-only RocksDB handle to consume events.
- **Pros**:
- No extra load on the OM process.
- Plugins can be added or removed at any time independent of all other plugins or OM.
- **Cons**:
- Process management complexity increases with number of plugins.
- Cannot do a local leader check if that is to be used as a run condition. Best option is probably to query metrics of the current host to check its leader status.
- Would need to add a new JMX endpoint if plugin metrics were to be added.
3. Run one process on each OM that internally runs all configured plugins
- Mostly the same pros and cons as the previous option.
- **Pros**:
- Less process management than the second option, while still removing all load from the OM.
- Plugins can be added, removed, or restarted independent of the OM, but not other plugins.
- **Cons**:
- Still need to manage a separate process with its own configuration and metrics.
- Cannot do a local leader check, still need to rely on the host OM's metrics.
For either approach, we could create a base plugin class that would handle most of the common tasks. The OM already supports dynamically loading plugins that implement a given interface like `IAccessAuthorizer`. We can use a similar model here:
- `OMEventListener` (or other name): The interface that all plugins must implement. It would probably just contain a single `run` method that would be started with the process and have complete autonomy over how it reads and processes events.
- `OMEventListenerBase` (or other name): An abstract class that provides a common starting implementation that most plugins would likely want to use for consuming events:
- Running a thread that fetches all the events from the DB from the last processed index, deserializing them, and invoking a callback to the implementation to consume them.
- Periodically persisting the last event index that was processed.
- This does not need to happen for every event due to at least once delivery.
- Persistence options include:
- Writing to a local file (harder to resume work on leader change)
- Writing the information to a file in Ozone
- Checking if the current node is the leader to determine whether the plugin should continue pushing events or sleep
- This failover mechanism also does not need to be exact due to at least once delivery. A follower may continue to deliver events before realizing it is no longer the leader without violating this protocol.
The biggest open item at the end of this discussion was whether to run the plugins within the OM as threads or as a separate process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2534896960
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
Review Comment:
how do we set regions for private deployments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3721334625
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512745719
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
I agree with this.
Creating a new RocksDB column family for a ledger table (which I guess is similar to a queue or distributed log like Apache BooKeeper) might not be desirable due to tombstones issue.
Implementing a queue-based semantic on top of RocksDB might not make sense (although there is a wiki like https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB) due to append only nature of RocksDB. Maybe we should use a persistent queue instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3527389956
@donalmag You can use this Feature Branch https://github.com/apache/ozone/tree/HDDS-13513_Event_Notification_FeatureBranch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501393336
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
(Just a passing comment) In the future, we can also support CloudEvents (https://cloudevents.io/) which is also supported by major cloud provider (AWS EventBridge, GCP Eventarc, Azure Event Grid) and ASF project like EventMesh (https://eventmesh.apache.org/docs/design-document/event-handling-and-integration/cloudevents/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3527411588
> use this Feature Branch https://github.com/apache/ozone/tree/HDDS-13513_Event_Notification_FeatureBranch
@swamirishi if you push and then delete a [feature branch](https://github.com/apache/ozone/activity?ref=HDDS-13513_Event_Notification_Design), please make sure to cancel the corresponding [workflow run](https://github.com/apache/ozone/actions/runs/19330005767).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2504317627
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
Nice, I haven't seen this before. With the plugin model we should be able to support any format as needed, and users can bring their own formats if those that end up shipping with Ozone don't cover their needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "donalmag (via GitHub)" <gi...@apache.org>.
donalmag commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2509694878
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
Updated doc to clarify points around retention and plugins running on OM.
@ChenSammi - the event capture only takes place if it is enabled via configuration.
The POC is available here - https://github.com/apache/ozone/pull/9012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2510854640
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
+2. Add a hook in the OMRequest execution workflow (post successful commit) to persist required events.
+3. Implement a plugin framework to run notification publishers.
+4. Implement a new background service for cleaning the events table, similar to KeyDeletingService, which operates
Review Comment:
No please don't add another ratis request and a background service in om unnecessarily. This can be done within the same om ratis transaction using rocksdb deleteRange API. as long as the column family key is going to ordered in the same order as the trasaction id. This would be a very cheap operation as this would just add another tombstone to rocksdb which should be ok
Please look at this implementation
https://github.com/apache/ozone/pull/8779/files#r2510853726
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
Review Comment:
For all the request we should also track the bucket layout(OBS/FSO). The consumer event may get some additional info on the kind of event. Behaviour of events like rename and delete are different might be a good info to track.
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
+2. Add a hook in the OMRequest execution workflow (post successful commit) to persist required events.
+3. Implement a plugin framework to run notification publishers.
+4. Implement a new background service for cleaning the events table, similar to KeyDeletingService, which operates
+in a deterministic manner across all OM's
+
+#### Plugin Framework
+
+Plugin Manager - spawns and supervises plugin threads.
+
+Base Plugin - Provides common functionality which can be re-used:
+1. Leader check.
+2. Read-only query on the RocksDb table.
+3. Offset tracking and persistence.
+4. MissedEvents notifications.
+
+It should be possible to run multiple plugins at the same time.
+
+Plugins should be configured such that the implementation can be loaded
+if provided on the classpath, similarly to ranger plugins which
+are configured as follows:
+
+```
+ xasecure.audit.destination.kafka=true
+ xasecure.audit.destination.kafka.classname=org.apache.ranger.audit.provider.kafka.KafkaAuditProvider
+ xasecure.audit.kafka.topic_name=ranger_audits
+ xasecure.audit.kafka.other_config_key=abc123
+```
+
+#### RocksDB Table
+
+The ledger will be stored as a RocksDb column family where the
+transaction id of the successful write operation is the key and the
+value is an object with the folliwng sample protobuf schema:
+
+```
+message CreateKeyOperationArgs {
+}
+
+message RenameKeyOperationArgs {
+ required string toKeyName = 1;
+}
+
+message DeleteKeyOperationArgs {
+}
+
+message CommitKeyOperationArgs {
+}
+
+message CreateDirectoryOperationArgs {
+}
+
+message CreateFileOperationArgs {
+ required bool isRecursive = 2;
+ required bool isOverwrite = 3;
+}
+
+message OperationInfo {
+
+ optional int64 trxLogIndex = 1;
+ required Type cmdType = 2; // Type of the command
+ optional string volumeName = 3;
+ optional string bucketName = 4;
+ optional string keyName = 5;
+ optional uint64 creationTime = 6;
+
+ optional CreateKeyOperationArgs createKeyArgs = 7;
+ optional RenameKeyOperationArgs renameKeyArgs = 8;
+ optional DeleteKeyOperationArgs deleteKeyArgs = 9;
+ optional CommitKeyOperationArgs commitKeyArgs = 10;
+ optional CreateDirectoryOperationArgs createDirectoryArgs = 11;
+ optional CreateFileOperationArgs createFileArgs = 12;
+}
+```
+
+## Performance
+
+Writes to the RocksDB table happen synchronously in the OM Commit path but are a single put operation.
+Deletes are to be executed by the OM in a separate thread ensuring the table is bounded to a specified limit.
Review Comment:
We have faced issues in DelegationToken implementation in the past because of the divergence. We don't want the event notification be another cause for this. Look at the comment above which has the way we can handle this.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
+
+Therefore to produce notifications using the standard S3 event
+notification semantics for FSO directory renames we would need to
+consider the trade-offs between compatibility with the normal S3
+semantics for renames vs a custom event type for directory renames.
+
+### most compatible approach
+
+We could introduce some additional processing before emitting notification
+events in the case of a directory rename which "gathers together" (prior
+to the change being committed to the DB) the child objects impacted by
+the directory rename and emits pairs of delete/create events for each
+key (as described above)
+
+Pros:
+- standard S3 event notification rename semantics
+
+Cons:
+- additional processing to pull together the events. This could mean an
+ unknown amount of additional processing for large directory renames.
+- could be a performance drag if performed on the leader
+
+### custom event type
+
+Conversely - we could opt to not try to be fully compliant with existing S3 event notification
+semantics since the schema was designed for non-hierarchical filesystems and
+instead create some custom event extension (e.g. ObjectRenamed:) and
+emit just a single event for directory renames which specifies only the parent
+paths impacted by the rename:
+
+e.g.
+```
+ eventName=ObjectReanmed:Reanmed, fromKey=myfiles, toKey=myfiles-RENAMED
+```
+
+.. it would then be up to the notification consumer to deal with the
+different rename event semantics (i.e. that only the parent names were
+notified and not the impacted child objects).
+
+This is the same semantics used in the HDFS inotify directory rename
+event (see below).
+
+Pros:
+- no additional processing when emitting events
+
+Cons:
+- non-standard S3 event notification semantics
+
+NOTE: directory rename is just one example of a hierarchical FSO
+operation which impacts child objects. There may be other Ozone
+hierarchical FSO operations which will need be catered for in a similar
+way (recursive delete?)
Review Comment:
@errose28 With OM leader execution designing for FSO the from path and toPath might not be defined. I am not sure how we would handle this in case of FSO? In leader execution the gate keeper would do path resolution parallely and only thing valid in that case would be parentId/keyName. I don't think we can really handle this correctly.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
Review Comment:
For all the request we should also track the bucket layout(OBS/FSO). The consumer event may get some additional info on the kind of event. Behaviour of events like rename and delete are different might be a good info to track.
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
+
+Therefore to produce notifications using the standard S3 event
+notification semantics for FSO directory renames we would need to
+consider the trade-offs between compatibility with the normal S3
+semantics for renames vs a custom event type for directory renames.
+
+### most compatible approach
+
+We could introduce some additional processing before emitting notification
+events in the case of a directory rename which "gathers together" (prior
+to the change being committed to the DB) the child objects impacted by
+the directory rename and emits pairs of delete/create events for each
+key (as described above)
+
+Pros:
+- standard S3 event notification rename semantics
+
+Cons:
+- additional processing to pull together the events. This could mean an
+ unknown amount of additional processing for large directory renames.
+- could be a performance drag if performed on the leader
+
+### custom event type
+
+Conversely - we could opt to not try to be fully compliant with existing S3 event notification
+semantics since the schema was designed for non-hierarchical filesystems and
+instead create some custom event extension (e.g. ObjectRenamed:) and
+emit just a single event for directory renames which specifies only the parent
+paths impacted by the rename:
+
+e.g.
+```
+ eventName=ObjectReanmed:Reanmed, fromKey=myfiles, toKey=myfiles-RENAMED
+```
+
+.. it would then be up to the notification consumer to deal with the
+different rename event semantics (i.e. that only the parent names were
+notified and not the impacted child objects).
+
+This is the same semantics used in the HDFS inotify directory rename
+event (see below).
+
+Pros:
+- no additional processing when emitting events
+
+Cons:
+- non-standard S3 event notification semantics
+
+NOTE: directory rename is just one example of a hierarchical FSO
+operation which impacts child objects. There may be other Ozone
+hierarchical FSO operations which will need be catered for in a similar
+way (recursive delete?)
Review Comment:
I believe we should spend some more time what events we want to emit in case of FSO. From what I understand order of event is very important maybe we have to send the entire hierarchy of objectIds witnessed while path resolution so within a ratis transaction batch all rename request we can figure out parents being renamed and the event added within a ratis batch has to be ordered.
For instance within a ratis batch say we have:
d1(O1)/d2(O2)/d3(O3)/F1(O4)
Now within a ratis transaction batch if I have 2 parallel transactions:
mv d1/d2 -> d1/d5
mv d1/d2/d3/F1 -> d1/d2/F1
then notification should be either
mv d1/d2/d3/F1 -> d1/d2/F1 and mv d1/d2 -> d1/d5
or
mv d1/d2 -> d1/d5 and mv d1/d5/d3/F1 -> d1/d5/F1
mv d1/d2 -> d1/d5 and mv d1/d2/d3/F1 -> d1/d2/F1 would be invalid
So a ratis txn batch should send the entire objectId hierarchy in the batch request to figure out this change and identify all the paths transformation this could get a bit complex I had experienced this first hand when we were implementing snapshot diff and we had scrapped the idea of making snaphshot diff order compliant but I believe this event notification design cannot be agnostic to order of events here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2714693516
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
We haven't benchmarked a POC but the design is such that there should be no conflict short of the read bandwdith of the NVMe holding RocksDB itself.
**event generation**
Events are committed to the same DB batch as their write request, so no overhead on the write path. The ledger is bounded by a configurable size so space consumption is fixed.
**read**
Unclear if this is referring to time it will take the plugins to read entries or impact to existing read requests. Plugins are processing entries completely async from within Ozone there will would be no impact on read requests and plugins can take as long as they need. I have not yet seen a case where RocksDB becomes the bottleneck on the read path on NVMe, especially across column families. We could benchmark a prototype but I would be surprised if we manage to saturate the disk's bandwidth with this.
**delete process status**
Also unclear whether this is referring to deleting entries from this table, or conflicts with key deletion in the OM. For deletion from this ledger, range deletes should handle any tombstone issues as mentioned above. There should be no conflicts with key deletion in OM for the same reason there are no conflicts on the read/write path.
**latency status of other OM operations, such as file list**
Same as above, there would only be a conflict if the plugins + active workload manage to saturate the NVMe read bandwidth, as RocksDB's reads should run in parallel, especially from different column families.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2711448398
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
Hi @errose28 , do we have any POC data about this part, that at an expected write workload, for example, 5k per second, what's the event generation, read, delete process status, and latency status of other OM operations, such as file list?
cc @umamaheswararao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3130093683
Thanks for adding a design for this. Based on the discussion in the community sync this morning here are some high level comments:
## Comments on the Specification
- Solidifying the schema first will help us work backwards to how we obtain the information
- We should decide whether we want to support a modified S3 schema, HDFS inotify schema, or one schema for FSO and one for OBS.
- We should define how this approach works with Ranger ACLS
- Likely this would just mean that if equivalent functionality was desired for Ranger ACLs, it would need to be incorporated to Ranger outside of Ozone. This is fine but it should be specified in the doc.
- The listener implemented for syncing may also be able to run the Ozone Ranger plugin itself and identify ACL changes that way.
- We need a way to define event retention. For example, if the notification service is down for x number of days or y number of transactions, all those before that period will be lost. Note that the service can be down for any arbitrary period of time.
- Are the notifications cluster-wide, or can they be scoped by bucket or object?
- Related to the previous point, how is access control implemented?
- Does the consumer of these events needs to authenticate with the cluster to make sure it has permission to see these changes, or is it purely a push model based on server side configs, where the configured endpoint is assumed to be trusted by the one configuring it?
- Some example configurations are called out for a Kafka based consumer, but it would probably be helpful to have a pluggable consumer model.
- See `IAccessAuthorizer` for one way to do this in the existing Ozone code, where the ACL authorizer can be loaded dynamically from a class provided by a config key.
- The "leader execution" project HDDS-11897 will change the format of the Ratis logs drastically. Instead of log contents like `rename key1 t-> key2` the logs will contain DB operations which will be difficult to map back to filesystem events, like `delete key1 from fileTable && put key2 in fileTable`.
- For future proofing we should make sure our approach works with both models
- If we need to add some extra metadata to the Ratis logs that is only read by the listener to support this, that might be ok.
## Comments on the design
- Using a listener node makes it difficult to ensure at-least-once delivery, since that listener node may need to install a Ratis snapshot at any time, losing the incremental Ratis log entries.
- The listener node is writing a full RocksDB which is never read. This will require a separate NVMe to keep up with the throughput of the main quorum, since applying to RocksDB is a sequential, blocking operation in Ratis. This incurs hardware overhead that is not really utilized for the task.
### Slight Alternative Approach: Co-locating a listener on each OM
I haven't totally worked out this proposal but wanted to put it out here since it seems to address some of the previous concerns. Instead of a full OM, the listener can be a small process that is co-located on each OM node. If its workload ends up being especially light, it may even be able to be a thread within the OM itself.
The main OM would be configured to move old Ratis log files to a backup directory instead of deleting them. This keeps its working directory clean and will not affect startup time due to a large number of files. I did a quick look through Ratis and it [doesn't look like this is supported currently](https://github.com/apache/ratis/blob/f21e350d4e7330d1b2f24001ac62040ebba205f4/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java#L483), but it could be added. The listener can read log entries from the backup dir, and then the main OM dir. As a listener, it will be notified of the cluster's apply index, which it can use to determine which log files correspond to valid events. It will also know the current leader through election events, so the instances running on followers can pause running. This listener can then push events to the plugged in consumers based on the Ratis logs, and purge them from the backup dir when the consumers have acke
d them. It does not need to consume the ratis logs that come through the Ratis listener API since it will use the local copies on the OM.
We would still need to hash out how the at-least-once delivery specification from Ozone to the consumer will fit with leader changes in this model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2498289836
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
If the connection to external target fails, will the events in DB still be removed by the retention policy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2510765866
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
+ "awsRegion":"us-east-1",
+ "eventTime":"2019-11-22T13:47:35.124724Z",
+ "eventName":"ObjectCreated:Put",
+ "userIdentity":{
+ "principalId":"tester"
+ },
+ "requestParameters":{
+ "sourceIPAddress":""
+ },
+ "responseElements":{
+ "x-amz-request-id":"503a4c37-85eb-47cd-8681-2817e80b4281.5330.903595",
+ "x-amz-id-2":"14d2-zone1-zonegroup1"
+ },
+ "s3":{
+ "s3SchemaVersion":"1.0",
+ "configurationId":"mynotif1",
+ "bucket":{
+ "name":"mybucket1",
+ "ownerIdentity":{
+ "principalId":"tester"
+ },
+ "arn":"arn:aws:s3:us-east-1::mybucket1",
+ "id":"503a4c37-85eb-47cd-8681-2817e80b4281.5332.38"
+ },
+ "object":{
+ "key":"myimage1.jpg",
+ "size":"1024",
+ "eTag":"37b51d194a7513e45b56f6524f2d51f2",
+ "versionId":"",
+ "sequencer": "F7E6D75DC742D108",
+ "metadata":[],
+ "tags":[]
+ }
+ },
+ "eventId":"",
+ "opaqueData":"me@example.com"
+ }
+]}
+```
+
+As we can see above: there are a number of boilerplate fields to inform us
+of various aspects of the completed operation but there are a few fundamental
+aspects to highlight;
+
+1. the "key" informs us of the key that the operation was performed on.
+
+2. the "eventName" informs us of the type of operation that was
+ performed. The 2 most notable eventNames are **ObjectCreated:Put** and
+ **ObjectRemoved:Deleted** which pertain to key creation and deletion respectively.
+
+3. operation specific fields can be included within the "object" sub-object (in
+ the above example we can see that "size" and "eTag" of the created object are included)
+
+## Applicability to Ozone
+
+For non-FSO Ozone buckets / operations there is a clear mapping between
+operations such as CreateKey / CommitKey / DeleteKey / RenameKey and the
+standard S3 event notification semantics.
+
+Examples:
+
+1. CommitKey could be mapped to a ObjectCreated:Put "/path/to/keyToCreate" notification event
+
+2. DeleteKey could be mapped to a ObjectRemoved:Deleted "/path/to/keyToDelete" notification event
+
+3. RenameKey (assuming a file based key) in standard S3 event noification semantics would produce 2 events:
+
+- a ObjectRemoved:Deleted event for the source path of the rename
+- a ObjectCreated:Put event for the destination path of the rename
+
+The challenge in adopting S3 Event notification semantics within Ozone
+would be in at least 2 areas:
+
+### 1. FSO hierarchical operations which impact multiple child keys
+
+Example: directory renames
+
+To illustrate with an example: lets say we have the following simple directory structure:
+
+```
+ /vol1/bucket1/myfiles/f1
+ /vol1/bucket1/myfiles/f2
+ /vol1/bucket1/myfiles/subdir/f1
+```
+
+If a user performs a directory rename such as:
+
+```
+ ozone fs -mv /vol1/bucket1/myfiles /vol1/bucket1/myfiles-RENAMED
+```
+
+Within standard S3 event notification semantics we would expect to see 6 notifications
+emitted in that case:
+
+```
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f1
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/f2
+ eventName=ObjectRemoved:Deleted, key=/vol1/bucket1/myfiles/subdir/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f1
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/f2
+ eventName=ObjectCreated:Put, key=/vol1/bucket1/myfiles-RENAMED/subdir/f1
+```
+
+However, with an approach of simply producing notifications based on Ratis
+state machine events then all we would have to go on from the
+RenameKeyRequest would be the fromKeyName and the toKeyName of the
+*parent* of the directory being renamed (and not the impacted child
+objects).
+
+Therefore to produce notifications using the standard S3 event
+notification semantics for FSO directory renames we would need to
+consider the trade-offs between compatibility with the normal S3
+semantics for renames vs a custom event type for directory renames.
+
+### most compatible approach
+
+We could introduce some additional processing before emitting notification
+events in the case of a directory rename which "gathers together" (prior
+to the change being committed to the DB) the child objects impacted by
+the directory rename and emits pairs of delete/create events for each
+key (as described above)
+
+Pros:
+- standard S3 event notification rename semantics
+
+Cons:
+- additional processing to pull together the events. This could mean an
+ unknown amount of additional processing for large directory renames.
+- could be a performance drag if performed on the leader
+
+### custom event type
+
+Conversely - we could opt to not try to be fully compliant with existing S3 event notification
+semantics since the schema was designed for non-hierarchical filesystems and
+instead create some custom event extension (e.g. ObjectRenamed:) and
+emit just a single event for directory renames which specifies only the parent
+paths impacted by the rename:
+
+e.g.
+```
+ eventName=ObjectReanmed:Reanmed, fromKey=myfiles, toKey=myfiles-RENAMED
+```
+
+.. it would then be up to the notification consumer to deal with the
+different rename event semantics (i.e. that only the parent names were
+notified and not the impacted child objects).
+
+This is the same semantics used in the HDFS inotify directory rename
+event (see below).
+
+Pros:
+- no additional processing when emitting events
+
+Cons:
+- non-standard S3 event notification semantics
+
+NOTE: directory rename is just one example of a hierarchical FSO
+operation which impacts child objects. There may be other Ozone
+hierarchical FSO operations which will need be catered for in a similar
+way (recursive delete?)
Review Comment:
I believe we should spend some more time what events we want to emit in case of FSO. From what I understand order of event is very important maybe we have to send the entire hierarchy of objectIds witnessed while path resolution so within a ratis transaction batch all rename request we can figure out parents being renamed and the event added within a ratis batch has to be ordered.
For instance within a ratis batch say we have:
d1(O1)/d2(O2)/d3(O3)/d4(O4)/F1(O5)
Now within a ratis transaction batch if I have 2 parallel transactions:
mv d1/d2 -> d1/d5
mv d1/d2/d3/d4/F1 -> d1/d2/d3/F1
then notification should be either
mv d1/d2/d3/F1 -> d1/d2/F1 and mv d1/d2 -> d1/d5
or
mv d1/d2 -> d1/d5 and mv d1/d5/d3/F1 -> d1/d5/F1
mv d1/d2 -> d1/d5 and mv d1/d2/d3/F1 -> d1/d2/F1 would be invalid
So a ratis txn batch should send the entire objectId hierarchy in the batch request to figure out this change and identify all the paths transformation this could get a bit complex I had experienced this first hand when we were implementing snapshot diff and we had scrapped the idea of making snaphshot diff order compliant but I believe this event notification design cannot be agnostic to order of events here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2534889156
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
Review Comment:
it would be nice to have 'Size to be reclaimed'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-4524643639
Might worth checking out TiCDC https://deepwiki.com/pingcap/ticdc/ (e.g. EventStore uses PebbleDB which Go rewrite for RocksDB, so we can check whether it fits our ledger DB schema)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-4898861381
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "gardenia (via GitHub)" <gi...@apache.org>.
gardenia commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3224601180
We are just updating the design doc based on the latest feedback and the discussion from last week's community call. I have realized I was left with some uncertainty over the ledger schema which I thought would be good to flesh out here before re-publishing the design doc.
My original strawman schema as discussed in last week's call:
```
message OperationInfo {
optional int64 trxLogIndex = 1;
optional string op = 2;
optional string volumeName = 3;
optional string bucketName = 4;
optional string keyName = 5;
optional uint64 creationTime = 6;
repeated hadoop.hdds.KeyValue extraArgs = 7;
}
```
@errose28 made a good point that the use of the extraArgs field (effectively a Map<String, String>) for miscellaneous fields (such as the rename "to" path or acl fields) would lead to a brittle schema management.
@errose28 you referenced a pattern where such things can be provided as a bunch of "optional" fields (for examples, as per OMRequest) but it wasn't clear exactly what you had in mind therefore I thought it would be useful to discuss a few possible intrepretations:
1. use the full request object in the ledger item schema, e.g. :
```
message OperationInfo {
required int64 trxLogIndex = 1;
required Type cmdType = 2; // Type of the command
required uint64 creationTime = 3;
optional CreateKeyRequest createKeyRequest = 4;
optional RenameKeyRequest renameKeyRequest = 5;
optional DeleteKeyRequest deleteKeyRequest = 6;
optional CommitKeyRequest commitKeyRequest = 7;
optional CreateDirectoryRequest createDirectoryRequest = 8;
optional CreateFileRequest createFileRequest = 9;
}
```
initially this is what I inferred that you meant but I may have misunderstood. At any rate - I'm not sure this is ideal - for a couple of reasons:
a. it pulls in the entire request object into the ledger which may be overkill
b. it is a bit loose about what it provides to consumers of the ledger and so those consumers could end up using fields they shouldn't and end up depending on them.
2. a less loose interpretation
```
message CreateKeyOperationArgs {
}
message RenameKeyOperationArgs {
required string toKeyName = 1;
}
message DeleteKeyOperationArgs {
}
message CommitKeyOperationArgs {
}
message CreateDirectoryOperationArgs {
}
message CreateFileOperationArgs {
required bool isRecursive = 2;
required bool isOverwrite = 3;
}
message OperationInfo {
optional int64 trxLogIndex = 1;
required Type cmdType = 2; // Type of the command
optional string volumeName = 3;
optional string bucketName = 4;
optional string keyName = 5;
optional uint64 creationTime = 6;
optional CreateKeyOperationArgs createKeyArgs = 7;
optional RenameKeyOperationArgs renameKeyArgs = 8;
optional DeleteKeyOperationArgs deleteKeyArgs = 9;
optional CommitKeyOperationArgs commitKeyArgs = 10;
optional CreateDirectoryOperationArgs createDirectoryArgs = 11;
optional CreateFileOperationArgs createFileArgs = 12;
}
```
This is effectively the same "optional" trick but rather than copy in the full request (from the OmRequest) we cherry-pick the fields which are exposed to the ledger via explicitly defined helper messages (RenameKeyOperationArgs, CreateFileOperationArgs etc) and so consumers have a clear contract as to what they can consume. This provides stronger typing around what extra fields are provided and this also makes schema changes simpler.
It could be argued that #1 gives more flexibility to consumers of the ledger but I feel like a more explicit contract (as per #2) is ultimately better and to keep it minimal to begin with makes sense. Therefore I would have a preference for #2.
But I would like to verify that I correctly understood your feedback @errose28 so please let me know what you (and others) think (or if you have a different suggestion).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2498289836
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
If the connection to external target fails, will the events in DB still be removed by the retention policy? What's the rule of retention policy will follow, size based, time window based, or other algorithms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2498289836
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
If the connection to external target fails, will the events in DB still be removed by the retention policy before send to targets? What's the rule of retention policy will follow, entry count based, time window based, or other algorithms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2534895102
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
+
+Notification events are produced as a list of JSON records.
+
+To illustrate we can look at a sample "create" event from the Ceph docs
+(https://docs.ceph.com/en/quincy/radosgw/notifications/#events):
+
+```json
+
+{"Records":[
+ {
+ "eventVersion":"2.1",
+ "eventSource":"ceph:s3",
Review Comment:
should it be "ozon:s3"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "donalmag (via GitHub)" <gi...@apache.org>.
donalmag commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3136206919
> ### Slight Alternative Approach: Co-locating a listener on each OM
> I haven't totally worked out this proposal but wanted to put it out here since it seems to address some of the previous concerns. Instead of a full OM, the listener can be a small process that is co-located on each OM node. If its workload ends up being especially light, it may even be able to be a thread within the OM itself.
>
> The main OM would be configured to move old Ratis log files to a backup directory instead of deleting them. This keeps its working directory clean and will not affect startup time due to a large number of files. I did a quick look through Ratis and it [doesn't look like this is supported currently](https://github.com/apache/ratis/blob/f21e350d4e7330d1b2f24001ac62040ebba205f4/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java#L483), but it could be added. The listener can read log entries from the backup dir, and then the main OM dir. As a listener, it will be notified of the cluster's apply index, which it can use to determine which log files correspond to valid events. It will also know the current leader through election events, so the instances running on followers can pause running. This listener can then push events to the plugged in consumers based on the Ratis logs, and purge them from the backup dir when the consumers have ac
ked them. It does not need to consume the ratis logs that come through the Ratis listener API since it will use the local copies on the OM.
>
> We would still need to hash out how the at-least-once delivery specification from Ozone to the consumer will fit with leader changes in this model.
@errose28 - Thanks for the feedback. This approach is interesting as it is similar to a separate POC we had developed internally. The reason we did not go with that approach was that we didn't think we could guarantee the entry in the ratis log was actually applied successfully without attempting to update the metadata itself first. Your comment seems to imply that should be possible though? Can you explain how we could verify this? My understanding is that the lastAppliedTxn is just the last txn the leader executed, there is no guarantee it was executed successfully - is that assumption incorrect? E.g. If the notify process is on txn 2 and the leader is on txn 10 how do we confirm which of txns 2->10 were applied successfully?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3887906935
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8871: HDDS-13513 Ozone Event Notification Design
URL: https://github.com/apache/ozone/pull/8871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2237282801
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
+
+# Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce a new OzoneManager node in the ratis ring in LISTENER mode.
+The node will be defined up-front in ozone-site.xml.
+This node will maintain metadata similar to the other OM nodes however it will generate notifications for events after they have been successfully commited.
+This node will not be able to become a leader, so the notification process will not impact client requests.
+
+
+
+#### Component
+
+Implementation of this feature requires changes to the OzoneManager:
+ - Implement an agnostic notification interface
+ - Add support for a LISTENER node
+ - Add a hook in the OMRequest exection flow to generate notifications
+
+#### Component
+
+Introduces a new field on OMAdminProtocol.proto to identify the notify node:
+
+```protobuf
+message OMNodeInfo {
+required string nodeID = 1;
+required string hostAddress = 2;
+required uint32 rpcPort = 3;
+required uint32 ratisPort = 4;
+optional NodeState nodeState = 5 [default=ACTIVE];
+optional bool isNotifyNode = 6 [default=false];
+}
+```
+
+## Performance
+
+While this is a synchronous only approach any latency between notification target and OM should not impact the performance of client requests as the notification does not run on the leader.
+
+## Configuration
+
+This is the configuration to enable event notification on node on1.
+
+```xml
+<property>
+ <name>ozone.event.notification.node</name>
+ <value>on1</value>
+</property>
+```
+Example configuration to provide a kafka instance:
+```xml
+
+<configration>
Review Comment:
Maybe as a future work, make it possible to configure notification on a per bucket basis, which is actually more AWS-like. That way each bucket owner can decide to send notification or send them to different kafka topics, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512732251
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
Do we have an estimation about how many events will be created and deleted in a period, say one day, if the feature is enabled, given OM average, and max two cases write IOPS? The write pattern of this column family is frequent write and deletion. We have seen that too many tombstone in RocksDB due to massive key and directory deletion slow down the OM and add latency to all OM requests which access RocksDB. I know we have the periodic compaction call to alleviate the impact, but I'm kind of afraid this new column family and it's write pattern will deepen this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512745719
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
I agree with this.
Creating a new RocksDB column family for a ledger table (which I guess is similar to a queue or distributed log like Apache BooKeeper) might not be desirable due to the ledger table.
Implementing a queue-based semantic on top of RocksDB might not make sense (although there is a wiki like https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB) due to append only nature of RocksDB. Maybe we should use a persistent queue instead,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2501393336
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
(Just a pass-through comment) In the future, we can also support CloudEvents (https://cloudevents.io/) which is also supported by major cloud provider (AWS EventBridge, GCP Eventarc, Azure Event Grid) and Apache project like EventMesh (https://eventmesh.apache.org/docs/design-document/event-handling-and-integration/cloudevents/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2240543847
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
Review Comment:
High availability and consistency.
It sounds like the design does not consider availability. If the Listener goes down there's no guarantee the events notification will be processed.
Consistency: does it guarantee in-order delivery of events? Any delivery latency assumptions or SLAs?
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,193 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occuring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+# Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+# Goals
+
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Not impact with performance of client requests.
+Guarantee at-least-once delivery.
+
+# Non-Goals
+
+Filtering of events or paths/buckets
+Persistent storage of notification messages
+Asynchronous delivery
+
+# Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce a new OzoneManager node in the ratis ring in LISTENER mode.
+The node will be defined up-front in ozone-site.xml.
+This node will maintain metadata similar to the other OM nodes however it will generate notifications for events after they have been successfully commited.
+This node will not be able to become a leader, so the notification process will not impact client requests.
+
+
+
+#### Component
+
+Implementation of this feature requires changes to the OzoneManager:
+ - Implement an agnostic notification interface
+ - Add support for a LISTENER node
+ - Add a hook in the OMRequest exection flow to generate notifications
+
+#### Component
+
+Introduces a new field on OMAdminProtocol.proto to identify the notify node:
+
+```protobuf
+message OMNodeInfo {
+required string nodeID = 1;
+required string hostAddress = 2;
+required uint32 rpcPort = 3;
+required uint32 ratisPort = 4;
+optional NodeState nodeState = 5 [default=ACTIVE];
+optional bool isNotifyNode = 6 [default=false];
+}
+```
+
+## Performance
+
+While this is a synchronous only approach any latency between notification target and OM should not impact the performance of client requests as the notification does not run on the leader.
+
+## Configuration
+
+This is the configuration to enable event notification on node on1.
+
+```xml
+<property>
+ <name>ozone.event.notification.node</name>
+ <value>on1</value>
+</property>
+```
+Example configuration to provide a kafka instance:
+```xml
+
+<configration>
+ <property>
+ <name>kafka.bootstrap.server</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.topic</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.saslUsername</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.saslPassword</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.saslMechanism</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.clientTlsCert</name>
+ <value></value>
+ </property>
+ <property>
+ <name>kafka.clientTlsKey</name>
+ <value></value>
+ </property>
+</configration>
+```
+
+### Metrics
+
+In order to better monitor the performance of the Notify Node the key metric here is how far we are behind the leader node, this can be achieved by comparing the lastAppliedTxn of each node.
+
+## Open questions
+
+### Notification Schema
+
+The S3 event notification schema https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types is based around immutable objects which cannot be renamed.
+In (AWS) S3 semantics renames are notified at the per-object level by sending a delete event on the old path and a create event on the new path.
+However, in Ozone objects can be renamed via the "RenameKey" ratis event which is triggered for both file and directory renames.
+In the context of a FSO directory rename we see a RenameKey ratis event for the directory name and not the granular objects.
+This would allow us to emit an event to denote that a path or directory was renamed but:
+
+It does not distinguish between file vs directory renames
+It does not allow us to emit cascading delete / create events for all the child objects being deleted/created as a result of a directory rename operation (as would be expected by traditional S3 event semantics)
+
+This leaves us with a gap with two potential solutions:
+
+1. Add a bespoke/“extended” event S3 type for RENAME operations which simply conveys that the key has been renamed (as per the semantics of RenameKey ratis callback) and which then leaves it up to the event consumer to deem the context (file vs directory) themselves (e.g. by stat-ing the target path). This is in-line with the semantics of HDFS inotify rename events:
+
+https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/hdfs/inotify/Event.RenameEvent.html
+
+Pros:
+* does not require any new processing on the OM side
+
+Cons:
+* non standard S3 event semantics
+
+2. The Ozone manager has to somehow pull together the child objects impacted by directory RENAME and emit appropriate remove/create events for the source/destination paths.
+
+Pros:
+* standard S3 event semantics
+
+Cons:
+* required additional processing on the OM side to gather the necessary child objects impacted and emit events for RENAME operations.
+
+As with RENAMEs there is no concept for ACL events in S3. If we stay with S3 semantics we could extend the schema similar to how Ceph has here: https://docs.ceph.com/en/latest/radosgw/s3-notification-compatibility/#event-types
+Or we if we took a HDFS Inotify view of the schema that already has inbuilt support for ACLs
+
+### At-least-once delivery
+
+At-least-once delivery cannot be guaranteed without some form of persistence, however we want to avoid persisting the notification messages themselves. One approach to achieving this is by "replaying" the proposals.
+If we persist the lastAppliedTxn after each notification is sent successfully, on a restart we could reset the lastAppliedTxn on the state machine and replay proposals in a "non-committal" mode in order to generate any missing notifications up to the current proposal.
Review Comment:
we do persist lastAppliedIndex in ratis log, and there's a callback to the statemachine implementation when a transaction is applied, so it sounds doable.
when we conceived the original design in HDDS-5984, we couldn't do it as S3 gateway is stateless. Making the logic inside the OM follower makes a lot of sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "donalmag (via GitHub)" <gi...@apache.org>.
donalmag commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2336134450
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
Review Comment:
@errose28 - I meant to mention this on the community call. We are suggesting that this ledger has to be "included in the snapshot process". The idea being that, we are relying on the ratis logs to update the ledger table, if a node loads a snapshot currently, it will not see all the ratis logs, therefore it's ledger will be missing events which the leader would have seen making it unable to fufil the 'at-least-once' guarantee. Have you any thoughts? The snapshot process doesn't seem readily extendible to include this, would this be the correct approach, or do you have any other thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2529600176
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
Ok, deleteRange should help. Previously I tried to explore implementing RocksDB queue, but it was abandoned. I'll wait for the performance number and whether the it meets the requirements when it's deployed to production by @donalmag and co.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r3276165022
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
Review Comment:
I have two thoughts:
1. we should support dynamic configuration(throught s3 endpoint) of destinations and only store the `CompletedRequestInfo` if there is destination existing(event notifications are not retroactive). and looks like we could support more complex configurtion, for each event notification:
- Event name,
- Prefix (optional)
- Suffix (optional)
- Destination (only one for one event notification)
ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html#:~:text=In%20the%20General,and%20Lambda.
so you might want to create a separate table to store these.
3. For the "retention policy" and "Offset tracking and persistence",
I think we should introduce a background service to mange the watermark of all downstream sink(eg. kafka destination).
the background service will basically loop on leader periodically:
1. check if we have been suspended and `isLeaderReady()`
1. `restart_txnid` = query smallest txn id in complete info with `iter.Seek();`
2. `confirmed_txnid` = min(txn id that all downstream have committed write/persist)
4. check the retention policy, assume we limit `max_record_keep_size` configured.
1. `reclaim_txnid = iter.key()`, after `iter.SeekToLast()` and run `iter.Prev()` for `max_record_keep_size` times
2. then `reclaim_range = [restart_txnid, max(reclaim_txnid, confirmed_txnid)]`
5. submit a raft request/response to delete `reclaim_range` with rocksdb `deleteRange`.
6. sleep a while and goto first step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-4528382047
> Might worth checking out TiCDC [deepwiki.com/pingcap/ticdc](https://deepwiki.com/pingcap/ticdc/) (e.g. EventStore uses PebbleDB which Go rewrite for RocksDB, so we can check whether it fits our ledger DB schema)
@ivandika3 Thanks for the info!
Looks like TiKV and TiCDC are designed to be disaggregated.
TiCDC pulls the CDC stream from TiKV, reading either from the Raft log or the MVCC history in RocksDB. Then TiCDC sends changes to downstream sink.
For Ozone, since we don't have builtin MVCC, we need to store the event into the ledger within the same key operation transaction in the same RocksDB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "donalmag (via GitHub)" <gi...@apache.org>.
donalmag commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2336134450
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,255 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
Review Comment:
@errose28 - I meant to mention this on the community call. We are suggesting that this ledger has to be "included in the snapshot process". The idea being that, we are relying on the ratis logs to update the ledger table, if a node loads a snapshot currently, it will not see all the ratis logs, therefore it's ledger will be missing events which the leader would have seen, making it unable to fufil the 'at-least-once' guarantee after a failover. The snapshot process doesn't seem readily extendible to include this, would this be the correct approach, or do you have any other thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3176844362
There was a healthy discussion on this topic today. Can any one summarize what we plan to do (or at least where we think we're converging onto?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3923932584
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2529600176
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
Ok, deleteRange should help. I'll wait for the performance number and whether the it meets the requirements when it's deployed to production by @donalmag and co.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "donalmag (via GitHub)" <gi...@apache.org>.
donalmag commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3136379975
FYI more detail on the event notification schema has been uploaded in a separate doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3141542793
> My understanding is that the lastAppliedTxn is just the last txn the leader executed, there is no guarantee it was executed successfully - is that assumption incorrect? E.g. If the notify process is on txn 2 and the leader is on txn 10 how do we confirm which of txns 2->10 were applied successfully?
Yes this is true. Actually the raft-log-only proposal would only work in the context of the new leader execution model in [HDDS-11897](https://issues.apache.org/jira/browse/HDDS-11897), where the commit index is guaranteed to have its corresponding operations applied to the state machine. We would then track the commit index to know where in the raft logs we need to generate events for. The current model will update the applied index even for invalid transactions that don't cause mutations like a directory already existing. The challenge is that you need an up to date OM DB to know whether or not a transaction will succeed. However this may require provisioning a whole OM Node with hardware just to push some json messages to an endpoint, and does not have at least once delivery in the presence of Ratis snapshot installs. This overhead would then become tech debt once the leader execution project finishes.
I think the high level design requirements we seem to be converging on are:
- A ledger of events
- Highly available persistence for this ledger to maintain at least once delivery
- Events will need to be persisted before returning to the client, although they may not need to be pushed to the consumers at this point.
- Minimal extra hardware provisioning
While using Raft logs seems to fit most of these requirements, it is not the only option if other Ratis constraints are getting in the way. Another option is that when the apply happens to the state machine, we write a ledger to the disk if that transaction indicates a successful modification. Basically the ledgers become part of the Ratis state machine, not the Ratis log. The notifier then consumes these ledgers and pushes them to the consumers without worrying about indices, because all ledgers represent successful operations. The notifier can prune these ledgers as it goes. Since we would likely need to add some metadata to the Ratis requests about operation type in HDDS-11897 for most event notification proposals I've seen so far, this should still work for both request models.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512745719
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
I agree with this.
Creating a new RocksDB column family for a ledger table (which I guess is similar to a queue or distributed log like Apache BooKeeper) might not be desirable due to tombstones issue which can make the OM DB size to increase by a lot.
Implementing a queue-based semantic on top of RocksDB might not make sense (although there is a wiki like https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB) due to append only nature of RocksDB. Maybe we should use a persistent queue instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512745719
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
I agree with this.
Creating a new RocksDB column family for a ledger table (which I guess is similar to a queue or distributed log like Apache BooKeeper) might not be desirable due to the ledger table.
Implementing a queue-based semantic on top of RocksDB might not make sense (although there is a wiki like https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB) due to append only nature of RocksDB. Maybe we should use a persistent queue instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #8871:
URL: https://github.com/apache/ozone/pull/8871#issuecomment-3514900980
Apologies for the late review. FYI, there is an alternative the design in Cross-region bucket replication (https://issues.apache.org/jira/browse/HDDS-12307) design docs (see "Phase 2: Near Real-time Live Replication").
The idea is to expose a subscribe endpoint in OM that will allow the client to do gRPC bidirectional streaming for all the updates. OM need to create a new log tailer for every subscription and send the updates to the client. This subscribed client can then store the events to its own persistent queue or send it to Kafka / RabbitMQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2512848224
##########
hadoop-hdds/docs/content/design/event-notifications.md:
##########
@@ -0,0 +1,257 @@
+---
+title: Event notification support in Ozone
+summary: Event notifications for all bucket/event types in ozone
+date: 2025-06-28
+jira: HDDS-13513
+status: design
+author: Donal Magennis, Colm Dougan
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Abstract
+
+Implement an event notification system for Apache Ozone, providing the ability for users to consume events occurring on the Ozone filesystem.
+This is similar to https://issues.apache.org/jira/browse/HDDS-5984 but aims to encapsulate all events and not solely S3 buckets.
+This document proposes a potential solution and discusses some of the challenges/open questions.
+
+## Introduction
+
+Apache Ozone does not currently provide the ability to consume filesystem events, similar to how HDFS does with Inotify or S3 with bucket notifications.
+These events are an integral part of integration with external systems to support real-time, scalable, and programmatic monitoring of changes in the data or metadata stored in Ozone.
+These external systems can use notifications of objects created/deleted to trigger data processing workflows, replication and monitoring alerts.
+
+### Goals
+
+Durable event log within each OM containing relevant OMRequest information for notification purposes.
+Plugin framework for publishers (e.g. Kafka/RabbitMQ, custom sinks) running in separate threads in the OM.
+Provide support for all events across the Ozone filesystem for FSO and non FSO buckets, including renames and changes to acls.
+Guarantee at-least-once delivery within a bounded retention period, with notification of "missed events" where applicable.
+Read-only access for plugins to notification table.
+
+### Non-Goals
+
+Exactly-once end-to-end semantics to external systems.
+Filtering of events or paths/buckets.
+Cross-OM consensus about what has been notified; co-ordination to be defined in the plugin e.g. write last notified position to a file in Ozone.
+Retrofitting historical events prior to feature enablement.
+
+### Supported OMRequests
+
+OMDirectoryCreateRequest
+OMKeyCommitRequest
+OMKeyDeleteRequest
+OMKeyRenameRequest
+OMKeyAddAclRequest
+OMKeyRemoveAclRequest
+OMKeySetAclRequest
+OMKeySetTimesRequest
+
+# Design
+
+## Overview
+
+Introduce an Event Notification Pipeline for Apache Ozone with two
+logical pieces:
+
+1. event data capture
+
+* OM captures the required details of selected OMRequest write
+ operations post metadata update and persists them to a dedicated RocksDB
+ completed operations "ledger" table keyed by the Ratis Txn Id
+* each OM independently produces items to its local ledger table. The
+ ledger table should be integrated into OM Snapshots so that all OM's
+ converge on the full set of required notifications.
+* a retention policy is to be implemented in order to clean up no longer required entries. This policy is bounded to a table size(number of events) which can be configurable.
+* event capture will only be enabled if enabled
+
+2. event data publishing
+
+* a plugin framework is exposed where plugins can consume the ledger
+ items in read-only fashion and process them as desired
+* Plugins will run inside the OM and should be cognisant of resource consumption i.e. memory/disk
+* all OMs will run the plugins but only the current leader OM will be
+ active
+* a base plugin implementation will provide common behaviour, including
+ read only iteration of new ledger items and flagging that events
+ have been "missed" since the consumer last requested them
+ leader OM will be active
+* a concrete plugin implementation will deal with publishing
+ notifications to external targets (Apache Kafka)
+
+### Components
+
+#### Ozone Manager
+
+Changes are required in the OzoneManager:
+1. Add a new RocksDB column family e.g. om_event_log.
Review Comment:
FYI, there is an alternative the CDC design in Cross-region bucket replication (https://issues.apache.org/jira/browse/HDDS-12307) design docs (see "Phase 2: Near Real-time Live Replication"). Please take a look whether it might be better.
The idea is to expose a subscribe endpoint in OM that will allow the client to do gRPC bidirectional streaming for all the updates. OM need to create a new log tailer for every subscription and send the updates to the client. This subscribed client can then store the events to its own persistent queue or send it to Kafka / RabbitMQ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13513 Ozone Event Notification Design [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on code in PR #8871:
URL: https://github.com/apache/ozone/pull/8871#discussion_r2534892542
##########
hadoop-hdds/docs/content/design/event-notification-schema.md:
##########
@@ -0,0 +1,396 @@
+---
+title: Event notification schema discussion
+summary: Event notifications schema discussion
+date: 2025-06-29
+jira: HDDS-13513
+status: design
+author: Colm Dougan, Donal Magennis
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+## Overview
+
+This document outlines the schema requirements for event notification
+within Ozone and discusses the suitability of 2 widely used event
+notification schemas (S3 and HDFS) as candidates to use as a basis for
+the transmission format for notifications within Ozone.
+
+# General schema requirements
+
+## File/Directory creation/modification
+
+event notifications should be raised to inform consumers of completed
+operations which modify the filesystem and specifically the requests:
+
+#### CreateRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateFileRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+nice to have fields:
+- overwrite
+- recursive
+
+#### CreateDirectoryRequest
+
+we should emit some **create** event
+
+required fields:
+- path (volume + bucket + key)
+- isfile
+
+#### CommitKeyRequest
+
+we should emit some **commit/close** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- data size
+- hsync?
+
+#### DeleteKeyRequest
+
+we should emit some **delete** event
+
+required fields:
+- path (volume + bucket + key)
+
+nice to have fields:
+- recursive (if known)
+
+### RenameKeyRequest
+
+we should emit some **rename** event
+
+required fields:
+- fromPath (volume + bucket + key)
+- toPath (volume + bucket + toKeyName)
+
+nice to have fields:
+- recursive (if known)
+- is directory (if known)
+
+NOTE: in the case of a FSO directory rename there is a dillema
+(discussed later in this document) as to whether we should emit a single
+event for a directory rename (specifying only the old/new directory names)
+or whether we should emit granular events for all the child objects impacted by
+the rename.
+
+## ACLs
+
+event notifications should be raised to inform consumers that ACL events
+have happened. The relevant requests are:
+
+* AddAclRequest
+* SetAclRequest
+* RemoveAclRequest
+
+The fields provided could vary based on the implementation complexity.
+
+Minimally we have a requirement that we be informed that "some ACL update
+happened" to a certain key (or prefix).
+
+Ideally the details would include the full context of the change made as
+per the request. (perhaps by mirroring the full request details as a JSON
+sub-object) e.g. :
+
+```json
+ ...
+
+ "acls": [
+ {
+ type: "GROUP",
+ name: "mygroup"
+ rights: "\000\001",
+ aclScope: "ACCESS",
+ }
+ ]
+```
+
+The precise details we would need to revisit with guidance from the
+community but this is just to set broad brush expectations.
+
+## SetTimes
+
+event notifications should be raised to inform consumers that
+mtime/atime has changed, as per **SetTimesRequest**
+
+# Transmission format
+
+This section discusses 2 widely used transmission formats for event
+notifiations (S3 and HDFS) and their suitability as candidates for
+adoption within Ozone.
+
+It is not assumed that these are the only options available but they are
+good examples to test against our requirements and discuss trade-offs.
+
+## 1. S3 Event Notification schema
+
+The S3 event notification schema:
+
+[https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html#supported-notification-event-types)
+
+has become a standard for change notifications in S3 compatible storage services such as S3 itself, Ceph, MinIO etc
Review Comment:
Can we adopt Cloud event format from the start?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org