You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2025/12/30 02:43:10 UTC
[I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
slfan1989 opened a new issue, #14945:
URL: https://github.com/apache/iceberg/issues/14945
### Background
In #14638, the discussion revolves around replacing MinIO in the Spark Quickstart due to the MinIO open-source repository entering maintenance mode and the presence of unpatched security issues. Currently, PR #14928 proposes using RustFS as a replacement. Based on this, I would like to propose a new solution: adding Apache Ozone as an optional storage backend in the Docker Compose Spark Quickstart example.
### Motivation
Apache Ozone is an actively maintained Apache top-level project (with the latest 2.0.0 release in 2025), fully open-source (Apache 2.0 license), and free from commercialization risks. It provides excellent S3 compatibility (s3a://) and native Hadoop file system semantics (ofs://), making it highly compatible with the Iceberg + Spark/Hadoop ecosystem. Ozone has been widely used in production environments for big data and AI workloads, offering outstanding scalability and integration advantages.
Adding Ozone as an option can better demonstrate the storage-agnostic nature of Iceberg, providing users with more local testing options and improving its overall flexibility and usability.
### Implementation Plan
The storage backend will be switched via an environment variable, for example:
```
STORAGE_BACKEND=minio|rustfs|ozone # Default is minio
```
The implementation includes:
- Adding Ozone service: Using the official apache/ozone Docker image in a single-node freestyle mode (this mode starts quickly and has low resource usage).
- Conditional Spark configuration: Dynamically configure spark-defaults.conf based on the selected backend, such as setting the warehouse path, S3 endpoint, etc.
- Updating the README: Provide clear instructions in the Quickstart README on how to enable Ozone as a storage backend, ensuring users can easily configure and test it.
### Next Steps
If the community finds this proposal valuable, I plan to submit a PR that includes the following:
- Update docker-compose.yml: Add the Ozone service and dynamically adjust service configurations based on the STORAGE_BACKEND environment variable.
- Modify the spark-defaults.conf template: Dynamically configure the relevant parameters (e.g., warehouse path, S3 endpoint) based on the selected storage backend.
- Update the README: Clearly explain how to switch between storage backends and provide steps for enabling Ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-5137764457
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14945: Add optional Apache Ozone backend to Spark Quickstart Docker demo
URL: https://github.com/apache/iceberg/issues/14945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-4986686672
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3709128134
Thanks @slfan1989 for the proposal and put the effort. Ozone 2.1.0 is closed to release now. It can be used for the initial version for docker image. I'm not sure if there is any dependency to adopt this version, for example, requiring a published Ozone 2.1.0 docker image first? cc @adoroszlai.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "umamaheswararao (via GitHub)" <gi...@apache.org>.
umamaheswararao commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3731301188
Great to see this effort @slfan1989.
As Ozone 2.1.0 is out already, we should go with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "dipankarmazumdar (via GitHub)" <gi...@apache.org>.
dipankarmazumdar commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3740879836
@SaketaChalamchala tested Ozone out with the "Iceberg Getting Started" notebook and worked great! Thanks for putting this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3700622965
> I'm fine having any implementation that follows proper licensing and is decently supported. I'm a little more afraid of having "optional" pieces in the quick start because i'd like to keep it as simple as possible. So I'd urge us to just pick either RustFS or Ozone (honestly whichever is simpler is probably best for the quickstart)
@RussellSpitzer Thanks for the feedback! I fully agree—we need to keep the Quickstart as straightforward as possible.
I'll go ahead and work on integrating Ozone using its official Docker images for the Spark Quickstart. Once both Ozone and RustFS options are implemented, we can compare them directly and collectively decide which one ends up being simpler and more suitable for new users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3737997593
Hi all, I submitted these patches to be able to use Ozone instead of MinIO in the quickstart envs. Please let me know your thoughts.
https://github.com/databricks/docker-spark-iceberg/pull/246
https://github.com/apache/iceberg/pull/15034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3700662678
I mean, with my Ozone hat on, I'd be thrilled to have Iceberg quick start integration example with Ozone, an Apache top level project so more folks can be aware Apache has an S3 compatible object store.
In the worse case if this is not accepted, we can have it it maintained at Ozone which is a WIP: https://ozone-site-v2.staged.apache.org/docs/user-guide/integrations/iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "myrust-go (via GitHub)" <gi...@apache.org>.
myrust-go commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3757854097
That's a good idea. RustFS and Ozone are both products with friendly licenses.
Let users choose for themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3698160199
@adoroszlai @ChenSammi @jojochuang @szetszwo @ayushtkn I would appreciate your thoughts and any suggestions you might have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3699956072
I'm fine having any implementation that follows proper licensing and is decently supported. I'm a little more afraid of having "optional" pieces in the quick start because i'd like to keep it as simple as possible. So I'd urge us to just pick either RustFS or Ozone (honestly whichever is simpler is probably best for the quickstart)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add optional Apache Ozone backend to Spark Quickstart Docker demo [iceberg]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on issue #14945:
URL: https://github.com/apache/iceberg/issues/14945#issuecomment-3741444531
@SaketaChalamchala Thank you very much for your contribution. I would give this a +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org