You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@iceberg.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2026/09/24 17:07:57 UTC

[PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

comphead opened a new pull request, #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273

## Which issue does this PR close?

- Closes #3272.

## What changes are included in this PR?

`quay.io/minio/minio` no longer allows anonymous pulls (see apache/datafusion#25705), so `make docker-up` fails and the `Start Docker containers` step of `Tests (default)` has been failing since 2026-09-24. DataFusion fixed the same breakage by switching to [RustFS](https://github.com/rustfs/rustfs) in apache/datafusion#25706, and PyIceberg moved to RustFS in apache/iceberg-python#3928. This PR does the same here.

In `dev/docker-compose.yaml`:

- The `minio` service becomes `rustfs`, running `rustfs/rustfs:1.0.0`. The `admin`/`password` credentials, the 9000 (S3) and 9001 (console) ports, and virtual-hosted-style access through `RUSTFS_SERVER_DOMAINS=rustfs` and the `*.rustfs` network aliases stay the same.
- The healthcheck uses `/health/ready`, which returns 200 only once storage and IAM are ready. `/health` is a liveness probe.
- The `mc` service becomes `create-buckets`. It reuses the RustFS image and creates `icebergdata`, `warehouse`, and `bucket1` with `curl --aws-sigv4`, as the DataFusion PR does. The `mc policy set public` calls are dropped because no test accesses the buckets anonymously.

The REST fixture, `dev/hms/core-site.xml`, and `dev/spark/spark-defaults.conf` now point at `http://rustfs:9000`. The test helpers are renamed to match: `get_minio_endpoint` becomes `get_rustfs_endpoint` and `ICEBERG_TEST_MINIO_ENDPOINT` becomes `ICEBERG_TEST_RUSTFS_ENDPOINT`.

The `minio://` examples in `crates/storage/opendal/src/lib.rs` are unchanged. They describe custom scheme support, not the test setup.

## Are these changes tested?

Covered by the existing integration tests, which run against RustFS in CI via `make docker-up`.

## AI Disclosure

Prepared with Claude Code (Claude Opus 5.5), modeled on apache/datafusion#25706.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu merged PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu commented on code in PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#discussion_r4097220185


##########
Makefile:
##########
@@ -112,7 +112,7 @@ site: install-mdbook
 
 # Docker targets for integration tests
 docker-up:
-	docker compose -f dev/docker-compose.yaml up -d --build --wait
+	docker compose -f dev/docker-compose.yaml up -d --build --wait --remove-orphans

Review Comment:
   in case previous container is still running, its now a different name so hence orphaned



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu commented on PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#issuecomment-5820301781

   we should just switch all the iceberg repos over, created a tracking issue: https://github.com/apache/iceberg/issues/18246


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "comphead (via GitHub)" <gi...@apache.org>.

comphead commented on PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#issuecomment-5822277066

   Thanks @kevinjqliu for the review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu commented on PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#issuecomment-5820160222

   > quay.io/minio/minio no longer allows anonymous pulls (see https://github.com/apache/datafusion/issues/25705)
   
   oh boy. thanks for the heads up. we just moved all minio to from docker hub to quay.io as a hot fix. now we gotta move everything again to rustfs 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.

Copilot commented on code in PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#discussion_r4096392885


##########
dev/docker-compose.yaml:
##########
@@ -25,51 +25,42 @@ networks:
 
 services:
   # =============================================================================
-  # MinIO - S3-compatible storage (shared by all tests)
+  # RustFS - S3-compatible storage (shared by all tests)
   # =============================================================================
-  minio:
-    image: quay.io/minio/minio:RELEASE.2025-05-24T17-08-30Z
+  rustfs:
+    image: rustfs/rustfs:1.0.0
     environment:
-      - MINIO_ROOT_USER=admin
-      - MINIO_ROOT_PASSWORD=password
-      - MINIO_DOMAIN=minio
-    hostname: minio
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=password
+      - RUSTFS_SERVER_DOMAINS=rustfs
+    hostname: rustfs
     networks:
       iceberg_test:
         # Add aliases for virtual-hosted style bucket access
         aliases:
-          - icebergdata.minio
-          - warehouse.minio
-          - bucket1.minio
+          - icebergdata.rustfs
+          - warehouse.rustfs
+          - bucket1.rustfs
     ports:
       - "9000:9000"
       - "9001:9001"
-    command: ["server", "/data", "--console-address", ":9001"]
     healthcheck:
-      test: ["CMD", "mc", "ready", "local"]
+      test: ["CMD", "curl", "-f", "http://localhost:9000/health/ready"]
       interval: 5s
       timeout: 5s
       retries: 5
 
-  # MinIO client - creates buckets for tests
-  mc:
+  # Creates buckets for tests
+  create-buckets:
     depends_on:
-      minio:
+      rustfs:
         condition: service_healthy
-    image: quay.io/minio/mc:RELEASE.2025-05-21T01-59-54Z
-    environment:
-      - AWS_ACCESS_KEY_ID=admin
-      - AWS_SECRET_ACCESS_KEY=password
-      - AWS_REGION=us-east-1
+    image: rustfs/rustfs:1.0.0
     entrypoint: >
       /bin/sh -c "
-        /usr/bin/mc alias set minio http://minio:9000 admin password;
-        /usr/bin/mc mb --ignore-existing minio/icebergdata;
-        /usr/bin/mc mb --ignore-existing minio/warehouse;
-        /usr/bin/mc mb --ignore-existing minio/bucket1;
-        /usr/bin/mc policy set public minio/icebergdata;
-        /usr/bin/mc policy set public minio/warehouse;
-        /usr/bin/mc policy set public minio/bucket1;
+        curl -fsS --aws-sigv4 aws:amz:us-east-1:s3 -u admin:password -X PUT http://rustfs:9000/icebergdata;
+        curl -fsS --aws-sigv4 aws:amz:us-east-1:s3 -u admin:password -X PUT http://rustfs:9000/warehouse;
+        curl -fsS --aws-sigv4 aws:amz:us-east-1:s3 -u admin:password -X PUT http://rustfs:9000/bucket1;

Review Comment:
   These commands use `;`, so a failed `curl -f` does not stop the shell: it continues to the success message and `tail -f /dev/null`. Compose therefore treats `create-buckets` as running even when a bucket was not created, and `make docker-up` can report success before tests fail later with misleading S3 errors. Chain the requests (or enable `set -e`) so provisioning failure makes the service fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] ci: replace MinIO with RustFS for integration tests [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.

kevinjqliu commented on PR #3273:
URL: https://github.com/apache/iceberg-rust/pull/3273#issuecomment-5820303822

   Thanks again @comphead 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org