You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "MonkeyCanCode (via GitHub)" <gi...@apache.org> on 2026/01/20 00:41:44 UTC
[PR] (feat) Add example for rustfs integration [polaris]
MonkeyCanCode opened a new pull request, #3482:
URL: https://github.com/apache/polaris/pull/3482
<!--
๐ Describe what changes you're proposing, especially breaking or user-facing changes.
๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more.
-->
Per discussion in https://lists.apache.org/thread/8o31ly7cd8ov70opjbtg630qlhrfl5yh, we would want to add support for RustFS integration. I follow the same layout as MinIO example for easy readability. If we do want to move forward with RustFS, I can then add more advance features such as assume role integration.
## Checklist
- [x] ๐ก๏ธ Don't disclose security issues! (contact security@apache.org)
- [x] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes #
- [x] ๐งช Added/updated tests with good coverage, or manually tested (and explained how)
- [x] ๐ก Added comments for complex logic
- [x] ๐งพ Updated `CHANGELOG.md` (if needed)
- [x] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode merged PR #3482:
URL: https://github.com/apache/polaris/pull/3482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709816744
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
That is correct with couple small syntax change where `mc` and `rc` are really for interacting with the s3-compatible storage while `aws` can do a lot more. Other than that, we will just need to set the endpoint-url for `aws` cli to `localhost:9000` then it should all works. But what is more interesting is how come there is no official image for `rc` and there are not much activities on that repo as well...even the `rc` repo is using `mc` for setting up buckets instead of using `rc` itself (but I did tested `rc` while I was working and this and it is simply drop-in replacement with change tool from `mc` to `rc`). While searching for this, I noticed there is a similar PR from Apache Iceberg for moving away from MinIO and replace with RustFS but they also ended up using `mc` for s3 interaction (https://github.com/apache/iceberg/pull/14928). Assuming `rc` will be more mature over time, it may be worth to keep `mc` for now? Or we can do docker-compose build or switch to aws cli is tha
t is more preferred and reuse-able.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708353875
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
Review Comment:
Yeah. Let me update this (we have couple more getting started example which are doing the same, so I will create a diff PR for them)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709793430
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
Review Comment:
With the feedback, we will keep this as we will be setting image tag for polaris be latest and pinned image versions for others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709181259
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
Sounds good. Will add a note around this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709174726
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
Review Comment:
So the current MinIO one is using latest image tag and have note regarding the build etc. But yeah, it may be beneficial to use official released one instead of local build. I will remove this and cleanup other ones later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709180130
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
Review Comment:
Yes. Same commented above. I will make change to use official released one then update other getting-started examples in a following up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708551420
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+ ```shell
+ docker compose -f getting-started/rustfs/docker-compose.yml up
Review Comment:
This is now resolved. So the problem appear to be from using multiple commands in healthcheck within podman compose:
```
"Healthcheck": {
"Test": [
"CMD",
"sh",
"-c",
"curl",
"-f",
"http://127.0.0.1:9000/health",
"\u0026\u0026",
"curl",
"-f",
"http://127.0.0.1:9001/rustfs/console/health"
],
"StartPeriod": 40000000000,
"Interval": 30000000000,
"Timeout": 10000000000,
"Retries": 3
},
```
Thus it triggered error as following:
```
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-01-20T08:13:51.901694752-06:00",
"End": "2026-01-20T08:13:51.940389164-06:00",
"ExitCode": 1,
"Output": "curl: try 'curl --help' or 'curl --manual' for more information\n"
}
]
},
```
To fix this, I switched to use `CMD-SHELL` instead so the shell form will avoid this issue. Also, this appears to be more preferred in podman as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709388834
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
Review Comment:
I'd prefer the latest Polaris release tag in examples too.
Developers can always adjust to a local tag, if they need to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708372886
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
So rustfs itself does have a lib for cli (https://github.com/rustfs/cli), however, there is no published image with only the tool (https://hub.docker.com/u/rustfs). Also, rustfs cli repo is also using `mc` as the official rustfs-init for creating bucket (https://github.com/rustfs/cli/blob/main/docker/docker-compose.yml). However, if we preferred (as mc and minio server won't get any new update), we can build our own client image with rustfs cli (`rc` in this case). What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708386267
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+ ```shell
+ docker compose -f getting-started/rustfs/docker-compose.yml up
Review Comment:
interesting as it works for me while I was writing this one last night, let me see why that is the case:
```
โ polaris git:(rustfs) docker compose -f getting-started/rustfs/docker-compose.yml up
WARN[0000] No services to build
[+] up 5/5
โ Network rustfs_default Created 0.0s
โ Container rustfs-rustfs-1 Created 0.0s
โ Container rustfs-setup_bucket-1 Created 0.0s
โ Container rustfs-polaris-1 Created 0.0s
โ Container rustfs-polaris-setup-1 Created 0.0s
Attaching to polaris-1, polaris-setup-1, rustfs-1, setup_bucket-1
Container rustfs-rustfs-1 Waiting
Container rustfs-rustfs-1 Waiting
rustfs-1 | Initializing data directories: /data
rustfs-1 | OBS log directory not configured and logs outputs to stdout
rustfs-1 | !!!WARNING: Using default RUSTFS_ACCESS_KEY or RUSTFS_SECRET_KEY. Override them in production!
rustfs-1 | Starting: /usr/bin/rustfs /data
rustfs-1 | RustFS Http API: http://172.18.0.2:9000 http://127.0.0.1:9000
rustfs-1 | RustFS Start Time: 2026-01-20 13:32:29
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.720648925Z","level":"WARN","fields":{"message":"Detected default credentials 'rustfsadmin:rustfsadmin', we recommend that you change these values with 'RUSTFS_ACCESS_KEY' and 'RUSTFS_SECRET_KEY' environment variables"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":171,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | Console WebUI Start Time: 2026-01-20 13:32:29
rustfs-1 | Console WebUI available at: http://172.18.0.2:9001/rustfs/console/index.html
rustfs-1 | Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.755828217Z","level":"WARN","fields":{"message":"config not found, start to init"},"target":"rustfs_ecstore::tier::tier","filename":"crates/ecstore/src/tier/tier.rs","line_number":490,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.757488175Z","level":"WARN","fields":{"message":"Configuration not found (Read the main configuration): Start initializing new configuration"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":137,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.759169967Z","level":"WARN","fields":{"message":"Configuration initialization complete (Read the main configuration)"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":139,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.778327675Z","level":"WARN","fields":{"message":"Global region is not set; attempting notification configuration for all buckets with an empty region."},"target":"rustfs::init","filename":"rustfs/src/init.rs","line_number":101,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.779977717Z","level":"WARN","fields":{"message":"main checkpoint file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-7a0b5a9f3ada404397601c2ff017fdca.json\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":154,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T13:32:29.779994508Z","level":"WARN","fields":{"message":"backup file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-7a0b5a9f3ada404397601c2ff017fdca.backup\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":172,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | RustFS server started successfully at [::]:9000, current time: 2026-01-20 13:32:29.780014092 UTC
Container rustfs-rustfs-1 Healthy
Container rustfs-rustfs-1 Healthy
setup_bucket-1 | Creating RustFS bucket...
Container rustfs-polaris-1 Waiting
polaris-1 | INFO exec -a "java" java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/quarkus-run.jar
polaris-1 | INFO running in /deployments
setup_bucket-1 | Added `pol` successfully.
setup_bucket-1 | Bucket created successfully `pol/bucket123`.
setup_bucket-1 | [2026-01-20 13:32:35 UTC] 0B bucket123/
setup_bucket-1 | Bucket setup complete.
polaris-1 | Listening for transport dt_socket at address: 5005
setup_bucket-1 exited with code 0
polaris-1 |
polaris-1 | @@
polaris-1 | @@
polaris-1 | @@@@
polaris-1 | @@@@@ @@ @@@ @@@@@@@@ @@@ @@@@@@@@ @@@ @@@ @@@@@@@@
polaris-1 | @@@@@@@@ @@@@ @@@@@ @@@ @@@ @@@@@ @@@@ @@@@ @@@ @@@ @@@
polaris-1 | @@@@@@@@ @@@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@@ @@@@@@@@@@ @@@@@@@
polaris-1 | @@@@@@@@@ @@@@@@@ @@@ @@@ @@@@@@@@ @@@ @@@ @@@@ @@@@@@@@@@ @@@@@@@
polaris-1 | @@@@@@@@@@@@@@@@@ @@@@@@@@@@ @@@ @@@@@@@@@@ @@@@ @@@@ @@@ @@@ @@@
polaris-1 | @@@@@@@@@@@@@@ @@@ @@@@ @@@ @@@ @@@@ @@@@@@@ @@@ @@@ @@@@@@@@
polaris-1 | @@@@@@@@@@@
polaris-1 | @@@@@@@@@@@@ @@@@@@@ @@@@@@@ @@@ @@@ @@@@@@@ @@@ @@@@@@
polaris-1 | @@@@@@@@@@@@@@@ @@@@@@@@@ @@@@@@@@@ @@@ @@@@@ @@@@@@@@ @@@ @@@ @@@
polaris-1 | @@@@@@@@@@@@@@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@@
polaris-1 | @@@@@@@@ @@@@@@@@ @@@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@@@@@ @@@ @@@@
polaris-1 | @@@@@@ @@@@@@ @@@ @@@@@@@@@ @@@ @@@@@@@@@@ @@@ @@@ @@@ @@@ @@@
polaris-1 | @@@@@ @@@@@@ @@@ @@@@@@@ @@@@@@@ @@@ @@@@ @@@ @@@ @@@ @@@@@@
polaris-1 | @@@ @@@@
polaris-1 | @@@@
polaris-1 | @@@
polaris-1 | @@
polaris-1 |
polaris-1 | Powered by Quarkus 3.29.4
polaris-1 | 2026-01-20 13:32:37,208 WARN [io.qua.config] [,] [,,,] (main) The "quarkus.log.file.enable" config property is deprecated and should not be used anymore.
polaris-1 | 2026-01-20 13:32:37,208 WARN [io.qua.config] [,] [,,,] (main) The "quarkus.log.console.enable" config property is deprecated and should not be used anymore.
polaris-1 | 2026-01-20 13:32:37,711 INFO [org.apa.pol.ser.con.ServiceProducers] [,] [,,,] (main) Bootstrapping realm(s) 'POLARIS', if necessary, from root credentials set provided via the environment variable POLARIS_BOOTSTRAP_CREDENTIALS or Java system property polaris.bootstrap.credentials ...
polaris-1 | 2026-01-20 13:32:37,830 INFO [org.apa.pol.ser.con.ServiceProducers] [,] [,,,] (main) Realm 'POLARIS' automatically bootstrapped, credentials taken from root credentials set provided via the environment variable POLARIS_BOOTSTRAP_CREDENTIALS or Java system property polaris.bootstrap.credentials, not printed to stdout.
polaris-1 | 2026-01-20 13:32:37,848 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) โ ๏ธ Production readiness checks failed! Check the warnings below.
polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - โ ๏ธ The current metastore is intended for tests only. Offending configuration option: 'polaris.persistence.type'.
polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - โ ๏ธ The realm context resolver is configured to map requests without a realm header to the default realm. Offending configuration option: 'polaris.realm-context.require-header'.
polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - โ ๏ธ A public key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.public-key-file'.
polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - โ ๏ธ A private key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.private-key-file'.
polaris-1 | 2026-01-20 13:32:37,850 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) Refer to https://polaris.apache.org/in-dev/unreleased/configuring-polaris-for-production for more information.
polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Apache Polaris Server (incubating) 1.3.0-incubating on JVM (powered by Quarkus 3.29.4) started in 2.406s. Listening on: http://0.0.0.0:8181. Management interface listening on http://0.0.0.0:8182.
polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Profile prod activated.
polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Installed features: [agroal, amazon-sdk-rds, cdi, hibernate-validator, jdbc-postgresql, micrometer, narayana-jta, oidc, opentelemetry, reactive-routes, rest, rest-jackson, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, vertx]
Container rustfs-polaris-1 Healthy
polaris-setup-1 | (1/2) Installing oniguruma (6.9.10-r0)
polaris-setup-1 | (2/2) Installing jq (1.8.1-r0)
polaris-setup-1 | Executing busybox-1.37.0-r30.trigger
polaris-setup-1 | OK: 14.6 MiB in 28 packages
polaris-1 | 2026-01-20 13:32:42,612 INFO [org.apa.pol.ser.con.PolarisIcebergObjectMapperCustomizer] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000001,POLARIS] [,,,] (executor-thread-1) Limiting request body size to 10485760 bytes
polaris-1 | 2026-01-20 13:32:42,641 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000001,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - - [20/Jan/2026:13:32:42 +0000] "POST /api/catalog/v1/oauth/tokens HTTP/1.1" 200 757
polaris-setup-1 | Creating catalog...
polaris-setup-1 | OK: 14.6 MiB in 28 packages
polaris-setup-1 |
polaris-setup-1 | Obtained access token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwb2xhcmlzIiwic3ViIjoicm9vdCIsImlhdCI6MTc2ODkxNTk2MiwiZXhwIjoxNzY4OTE5NTYyLCJqdGkiOiI3OTgyYzk2OS1mYmUxLTQyM2ItYTI3NS1iZGY1MTJjYmY3ZDMiLCJhY3RpdmUiOnRydWUsImNsaWVudF9pZCI6InJvb3QiLCJwcmluY2lwYWxJZCI6MSwic2NvcGUiOiJQUklOQ0lQQUxfUk9MRTpBTEwifQ.wJ0dW7jdKmNJ_qCkWNd2ImTJhuw66ZaHbnYJAIpoYUpra5rj3LvWnoNpbXw_c4bvxyFvcDwgdCZx8PuXOlF1DDSDaM3ZMMxBe852J_rHKjxomN-Fz_ndXLWgmjibjgO5SbW1qTvOqzOpDXcCgwtlrtZYHi2sL27W5HQb35ieySLT6GH6BhypBT1NbWXtOuITBBRJzqSYO12oOWi35SBu-aKinIeUQytkrdgx8ZtzIVOY4OsZzERfJaQm_IjBsPh4diXQr2teeGU-kSMIEgHuqUaa-ldD6_zwv1JxR45lYbooE8kcaIxWl4xDUM6SvcJ3feQQ92Z5j7UrQlfc_VlE3w
polaris-setup-1 | STORAGE_LOCATION is set to 's3://bucket123'
polaris-setup-1 | Using StorageType: S3
polaris-setup-1 |
polaris-setup-1 | Creating a catalog named quickstart_catalog in realm POLARIS...
polaris-setup-1 | { "catalog": { "name": "quickstart_catalog", "type": "INTERNAL", "readOnly": false, "properties": { "default-base-location": "s3://bucket123" }, "storageConfigInfo": {"storageType":"S3", "endpoint":"http://localhost:9000", "endpointInternal":"http://rustfs:9000", "pathStyleAccess":true} } }
polaris-setup-1 | * Host polaris:8181 was resolved.
polaris-setup-1 | * IPv6: (none)
polaris-setup-1 | * IPv4: 172.18.0.4
polaris-setup-1 | * Trying 172.18.0.4:8181...
polaris-setup-1 | * Established connection to polaris (172.18.0.4 port 8181) from 172.18.0.3 port 38416
polaris-setup-1 | * using HTTP/1.x
polaris-setup-1 | > POST /api/management/v1/catalogs HTTP/1.1
polaris-setup-1 | > Host: polaris:8181
polaris-setup-1 | > User-Agent: curl/8.17.0
polaris-setup-1 | > Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwb2xhcmlzIiwic3ViIjoicm9vdCIsImlhdCI6MTc2ODkxNTk2MiwiZXhwIjoxNzY4OTE5NTYyLCJqdGkiOiI3OTgyYzk2OS1mYmUxLTQyM2ItYTI3NS1iZGY1MTJjYmY3ZDMiLCJhY3RpdmUiOnRydWUsImNsaWVudF9pZCI6InJvb3QiLCJwcmluY2lwYWxJZCI6MSwic2NvcGUiOiJQUklOQ0lQQUxfUk9MRTpBTEwifQ.wJ0dW7jdKmNJ_qCkWNd2ImTJhuw66ZaHbnYJAIpoYUpra5rj3LvWnoNpbXw_c4bvxyFvcDwgdCZx8PuXOlF1DDSDaM3ZMMxBe852J_rHKjxomN-Fz_ndXLWgmjibjgO5SbW1qTvOqzOpDXcCgwtlrtZYHi2sL27W5HQb35ieySLT6GH6BhypBT1NbWXtOuITBBRJzqSYO12oOWi35SBu-aKinIeUQytkrdgx8ZtzIVOY4OsZzERfJaQm_IjBsPh4diXQr2teeGU-kSMIEgHuqUaa-ldD6_zwv1JxR45lYbooE8kcaIxWl4xDUM6SvcJ3feQQ92Z5j7UrQlfc_VlE3w
polaris-setup-1 | > Accept: application/json
polaris-setup-1 | > Content-Type: application/json
polaris-setup-1 | > Polaris-Realm: POLARIS
polaris-setup-1 | > Content-Length: 341
polaris-setup-1 | >
polaris-setup-1 | } [341 bytes data]
polaris-setup-1 | * upload completely sent off: 341 bytes
polaris-1 | 2026-01-20 13:32:43,030 INFO [org.apa.pol.ser.adm.PolarisServiceImpl] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002,POLARIS] [,,,] (executor-thread-1) Created new catalog class PolarisCatalog {
polaris-1 | class Catalog {
polaris-1 | type: INTERNAL
polaris-1 | name: quickstart_catalog
polaris-1 | properties: class CatalogProperties {
polaris-1 | {default-base-location=s3://bucket123}
polaris-1 | defaultBaseLocation: s3://bucket123
polaris-1 | }
polaris-1 | createTimestamp: 1768915963026
polaris-1 | lastUpdateTimestamp: 0
polaris-1 | entityVersion: 1
polaris-1 | storageConfigInfo: class AwsStorageConfigInfo {
polaris-1 | class StorageConfigInfo {
polaris-1 | storageType: S3
polaris-1 | allowedLocations: [s3://bucket123]
polaris-1 | }
polaris-1 | roleArn: null
polaris-1 | externalId: null
polaris-1 | userArn: null
polaris-1 | region: null
polaris-1 | endpoint: http://localhost:9000
polaris-1 | stsEndpoint: null
polaris-1 | stsUnavailable: null
polaris-1 | endpointInternal: http://rustfs:9000
polaris-1 | pathStyleAccess: true
polaris-1 | }
polaris-1 | }
polaris-1 | }
polaris-1 | 2026-01-20 13:32:43,035 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - root [20/Jan/2026:13:32:43 +0000] "POST /api/management/v1/catalogs HTTP/1.1" 201 354
polaris-setup-1 | < HTTP/1.1 201 Created
polaris-setup-1 | < Content-Type: application/json;charset=UTF-8
polaris-setup-1 | < content-length: 354
polaris-setup-1 | < X-Request-ID: 86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002
polaris-setup-1 | <
polaris-setup-1 | { [354 bytes data]
polaris-setup-1 | * Connection #0 to host polaris:8181 left intact
polaris-setup-1 | {"type":"INTERNAL","name":"quickstart_catalog","properties":{"default-base-location":"s3://bucket123"},"createTimestamp":1768915963026,"lastUpdateTimestamp":0,"entityVersion":1,"storageConfigInfo":{"endpoint":"http://localhost:9000","endpointInternal":"http://rustfs:9000","pathStyleAccess":true,"storageType":"S3","allowedLocations":["s3://bucket123"]}}
polaris-setup-1 | Done.
polaris-setup-1 | Extra grants...
polaris-setup-1 | % Total % Received % Xferd Average Speed Time Time Time Current
polaris-setup-1 | Dload Upload Total Spent Left Speed
polaris-1 | 2026-01-20 13:32:43,047 INFO [org.apa.pol.ser.adm.PolarisServiceImpl] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000003,POLARIS] [,,,] (executor-thread-1) Adding grant class AddGrantRequest {
polaris-1 | grant: class CatalogGrant {
polaris-1 | class GrantResource {
polaris-1 | type: catalog
polaris-1 | }
polaris-1 | privilege: CATALOG_MANAGE_CONTENT
polaris-1 | }
polaris-1 | } to catalogRole catalog_admin in catalog quickstart_catalog
polaris-1 | 2026-01-20 13:32:43,062 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000003,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - root [20/Jan/2026:13:32:43 +0000] "PUT /api/management/v1/catalogs/quickstart_catalog/catalog-roles/catalog_admin/grants HTTP/1.1" 201 -
100 56 0 0 100 56 0 2274 --:--:-- --:--:-- --:--:-- 2333
polaris-setup-1 | Done.
polaris-setup-1 exited with code 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on PR #3482:
URL: https://github.com/apache/polaris/pull/3482#issuecomment-3774636862
> LGTM assuming current comment threads are resolved :) Thanks, @MonkeyCanCode !
Thanks for the quick review @dimas-b and @snazy. I had addressed all of the comments. i would like to have this up for couple more days so more people can review. I do see RustFS being a viable replacement for MinIO. However, there are couple concerns on my end as well as called out by community from the mailing list:
1. Will RustFS be the next "MinIO" in couple of months/years?
2. There are a bit out of documents for RustFS for certain areas (such as STS...but it doesn't appear they support it but would need more testing for validation if we want to move forward with it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708484239
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+ ```shell
+ docker compose -f getting-started/rustfs/docker-compose.yml up
Review Comment:
So compare the output you have and the one I shared above, one thing I am seeing diff is I am using docker-compose and you are using podman-compose. While trying to reproduce this issue on my end, i am able to get to the same state as what you mentioned above with podman-compose. Let me take a closer look to see why podman is not happy with this. I will get back a bit later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709101571
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
Review Comment:
Is this step needed? Looks like the `docker-compose.yml` would pull 1.3.0 and not use the built image.
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
Review Comment:
Nit: If the above step isn't needed, this `2.` ordered list item becomes superfluous.
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
Nah - I'm okay leaving `mc` for now. Once they publish their `rc` as a container image, we can always switch to that one. This is rather a "demonstration".
Might be worth to add a comment to the compose file that `mc` is from Minio and won't receive any updates and `rc` should be used in production setups.
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+ ```shell
+ docker compose -f getting-started/rustfs/docker-compose.yml up
Review Comment:
Thanks! It works on my machine w/ podman now.
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
Review Comment:
Should there a link to RustFS? Maybe here or in a final "Further information" section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709176817
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
Review Comment:
Good thinking. Let me add it later tonight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709406877
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
`aws` CLI should work too, I assume, but I'm fine with `mc`. I also saw it in docs related to RustFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "snazy (via GitHub)" <gi...@apache.org>.
snazy commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2707897285
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
+ ports:
+ - "9000:9000" # S3 API port
+ - "9001:9001" # Console port
+ environment:
+ RUSTFS_ACCESS_KEY: rustfsadmin
+ RUSTFS_SECRET_KEY: rustfsadmin
+ RUSTFS_VOLUMES: /data
+ RUSTFS_ADDRESS: ":9000"
+ RUSTFS_CONSOLE_ENABLE: "true"
+ RUSTFS_CONSOLE_ADDRESS: ":9001"
+ healthcheck:
+ test:
+ [
+ "CMD",
+ "sh", "-c",
+ "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+ ]
+ interval: 30s
+ timeout: 10s
+ retries: 3
+ start_period: 40s
+
+ polaris:
+ image: apache/polaris:latest
+ ports:
+ # API port
+ - "8181:8181"
+ # Optional, allows attaching a debugger to the Polaris JVM
+ - "5005:5005"
+ depends_on:
+ rustfs:
+ condition: service_healthy
+ environment:
+ JAVA_DEBUG: true
+ JAVA_DEBUG_PORT: "*:5005"
+ AWS_REGION: us-west-2
+ AWS_ACCESS_KEY_ID: rustfsadmin
+ AWS_SECRET_ACCESS_KEY: rustfsadmin
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t
+ polaris.realm-context.realms: POLARIS
+ quarkus.otel.sdk.disabled: "true"
+ healthcheck:
+ test: ["CMD", "curl", "http://localhost:8182/q/health"]
+ interval: 2s
+ timeout: 10s
+ retries: 10
+ start_period: 10s
+
+ setup_bucket:
+ image: quay.io/minio/mc:latest
Review Comment:
Wonder why `mc` is being used here. Doesn't Rustfs provide a tool for this as well?
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
+
+2. Start the docker compose group by running the following command from the root of the repository:
+
+ ```shell
+ docker compose -f getting-started/rustfs/docker-compose.yml up
Review Comment:
This one fails for me after a while:
```
docker compose -f getting-started/rustfs/docker-compose.yml up
>>>> Executing external compose provider "/usr/libexec/docker/cli-plugins/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<
[+] Running 17/17
โ setup_bucket Pulled 3.1s
โ b83ce1c86227 Already exists 0.0s
โ b7c1efd1d2b4 Download complete 0.7s
โ b268f447c871 Download complete 0.0s
โ f29310fc4a8a Download complete 0.0s
โ 9404e0df6bd3 Download complete 1.1s
โ 4a93952ab9ce Download complete 0.1s
โ c2ad77420d33 Download complete 0.0s
โ rustfs Pulled 6.7s
โ 4f4fb700ef54 Already exists 0.0s
โ 1074353eec0d Already exists 0.0s
โ 1d1dee9383a6 Download complete 0.0s
โ 82f2cf490642 Download complete 0.0s
โ d3ec0616f053 Download complete 0.0s
โ efb5b00da6a9 Download complete 0.3s
โ 981fcf90905c Download complete 5.5s
โ c83e944dc36e Download complete 0.0s
[+] Running 5/5
โ Network rustfs_default Created 0.0s
โ Container rustfs-rustfs-1 Created 0.7s
โ Container rustfs-setup_bucket-1 Created 0.0s
โ Container rustfs-polaris-1 Created 0.0s
โ Container rustfs-polaris-setup-1 Created 0.0s
Attaching to polaris-1, polaris-setup-1, rustfs-1, setup_bucket-1
rustfs-1 | Initializing data directories: /data
rustfs-1 | OBS log directory not configured and logs outputs to stdout
rustfs-1 | !!!WARNING: Using default RUSTFS_ACCESS_KEY or RUSTFS_SECRET_KEY. Override them in production!
rustfs-1 | Starting: /usr/bin/rustfs /data
rustfs-1 | RustFS Http API: http://10.89.0.4:9000 http://127.0.0.1:9000
rustfs-1 | RustFS Start Time: 2026-01-20 11:14:43
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.212466879Z","level":"WARN","fields":{"message":"Detected default credentials 'rustfsadmin:rustfsadmin', we recommend that you change these values with 'RUSTFS_ACCESS_KEY' and 'RUSTFS_SECRET_KEY' environment variables"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":171,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | Console WebUI Start Time: 2026-01-20 11:14:43
rustfs-1 | Console WebUI available at: http://10.89.0.4:9001/rustfs/console/index.html
rustfs-1 | Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.215548374Z","level":"WARN","fields":{"message":"config not found, start to init"},"target":"rustfs_ecstore::tier::tier","filename":"crates/ecstore/src/tier/tier.rs","line_number":490,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.21644797Z","level":"WARN","fields":{"message":"Configuration not found (Read the main configuration): Start initializing new configuration"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":137,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.217214776Z","level":"WARN","fields":{"message":"Configuration initialization complete (Read the main configuration)"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":139,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.220595899Z","level":"WARN","fields":{"message":"Global region is not set; attempting notification configuration for all buckets with an empty region."},"target":"rustfs::init","filename":"rustfs/src/init.rs","line_number":101,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.220696059Z","level":"WARN","fields":{"message":"main checkpoint file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-a00d591b5b1a4f3e9731b123df370860.json\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":154,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | {"timestamp":"2026-01-20T11:14:43.220705629Z","level":"WARN","fields":{"message":"backup file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-a00d591b5b1a4f3e9731b123df370860.backup\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":172,"threadName":"main","threadId":"ThreadId(1)"}
rustfs-1 | RustFS server started successfully at [::]:9000, current time: 2026-01-20 11:14:43.220719429 UTC
dependency failed to start: container rustfs-rustfs-1 is unhealthy
Error: executing /usr/libexec/docker/cli-plugins/docker-compose -f getting-started/rustfs/docker-compose.yml up: exit status 1
```
The health-check command itself seems to work fine though:
```
$ docker container exec c169ceab8242 sh -c 'curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 111 100 111 0 0 285347 0 --:--:-- --:--:-- --:--:-- 111000
{"service":"rustfs-endpoint","status":"ok","timestamp":"2026-01-20T11:20:07.728989111+00:00","version":"0.0.5"} % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 204 100 204 0 0 532637 0 --:--:-- --:--:-- --:--:-- 204000
{"details":{"iam":{"status":"connected"},"storage":{"status":"connected"}},"service":"rustfs-console","status":"ok","timestamp":"2026-01-20T11:20:07.732014419+00:00","uptime":1768908007,"version":"0.0.5"}
```
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
Review Comment:
Can you add a "real" version here and below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2708507682
##########
getting-started/rustfs/docker-compose.yml:
##########
@@ -0,0 +1,116 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+services:
+
+ rustfs:
+ image: rustfs/rustfs:latest
Review Comment:
This is now resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on code in PR #3482:
URL: https://github.com/apache/polaris/pull/3482#discussion_r2709780540
##########
getting-started/rustfs/README.md:
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Getting Started with Apache Polaris and RustFS
+
+## Overview
+
+This example uses RustFS as a storage provider with Polaris.
+
+Spark is used as a query engine. This example assumes a local Spark installation.
+See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup.
+
+## Starting the Example
+
+1. Build the Polaris server image if it's not already present locally:
+
+ ```shell
+ ./gradlew \
+ :polaris-server:assemble \
+ :polaris-server:quarkusAppPartsBuild --rerun \
+ -Dquarkus.container-image.build=true
+ ```
Review Comment:
That makes sense. Lets pinned other things which we don't to a fixed version and keep polaris with latest tag. By doing so, if user didn't build locally, it will get from dockerhub and if user need to run some new testings, a local rebuild then rerun the docker-compose will take care this. That being said, above instruction will be keep as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #3482:
URL: https://github.com/apache/polaris/pull/3482#issuecomment-3775499567
Looks like RustFS is actively maintained, a lot of commits in last a few weeks https://github.com/rustfs/rustfs/commits/main/?after=ae50760fcc5703a1e4e1c05ab656fc0fff366f34+104. It does seem a good candidate of MinIO alternatives.
> Will RustFS be the next "MinIO" in couple of months/years?
I assume it's really hard to answer it. This blog provides a hint on their vision, https://rustfs.dev/hello-world-rustfs-and-our-journey-to-democratize-storage/, which seems promising on OSS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #3482:
URL: https://github.com/apache/polaris/pull/3482#issuecomment-3774762402
> There are a bit out of documents for RustFS for certain areas (such as STS...but it doesn't appear they support it [...]
This guide uses `vended-credentials`, so I assume RustFS supports STS... otherwise how does the example work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (feat) Add example for rustfs integration [polaris]
Posted by "MonkeyCanCode (via GitHub)" <gi...@apache.org>.
MonkeyCanCode commented on PR #3482:
URL: https://github.com/apache/polaris/pull/3482#issuecomment-3774883089
> > There are a bit out of documents for RustFS for certain areas (such as STS...but it doesn't appear they support it [...]
>
>
>
> This guide uses `vended-credentials`, so I assume RustFS supports STS... otherwise how does the example work?
Yes, this is fixed recently (https://github.com/rustfs/rustfs/issues/1337) but no where I can find the doc around this when I was using older image a while back and found this issue and switched to the latest patched image as part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org