You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "colinmarc (via GitHub)" <gi...@apache.org> on 2025/07/11 18:52:50 UTC
[PR] feat: bump datafusion to 48 [iceberg-rust]
colinmarc opened a new pull request, #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501
This did not require very many changes!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3272544265
we can potentially do something like this
https://github.com/guacsec/guac/blob/3aa915a571f2c3a097aa236144eb990b46a7fbb6/Makefile#L275-L281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3272527704
looks like the protoc requirement is from
`datafusion-sqllogictest` -> `datafusion-substrait` -> `substrait`
Heres the original error message:
```
error: failed to run custom build command for `substrait v0.56.0`
Caused by:
process didn't exit successfully: `/home/runner/work/iceberg-rust/iceberg-rust/target/debug/build/substrait-9f341031683ef1a7/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=FORCE_REBUILD
cargo:rerun-if-changed=substrait
cargo:rerun-if-changed=substrait/text/simple_extensions_schema.yaml
cargo:rerun-if-changed=substrait/proto/substrait/type.proto
cargo:rerun-if-changed=substrait/proto/substrait/extensions/extensions.proto
cargo:rerun-if-changed=substrait/proto/substrait/extended_expression.proto
cargo:rerun-if-changed=substrait/proto/substrait/algebra.proto
cargo:rerun-if-changed=substrait/proto/substrait/plan.proto
cargo:rerun-if-changed=substrait/proto/substrait/function.proto
cargo:rerun-if-changed=substrait/proto/substrait/parameterized_types.proto
cargo:rerun-if-changed=substrait/proto/substrait/type_expressions.proto
cargo:rerun-if-changed=substrait/proto/substrait/capabilities.proto
--- stderr
Error: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:21: build] Error 101
```
> If this is the case, I would suggest to also add a step in our make file to fix it?
I uninstalled `protoc` locally with `brew uninstall protobuf` and verify that `make build` fails locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3267499940
I tried regenerating the Cargo.lock file, lets see if this will resolve CI :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3264005662
summarizing the slack thread:
we are currently blocked due to MSRV versions.
Datafusion 48 requires aws-config 1.6.2 which requires rust 1.86.
We're currently using rust 1.85.
Typically we bump the MSRV for a new release of iceberg-rust, https://github.com/apache/iceberg-rust?tab=readme-ov-file#supported-rust-version
I started the PR to upgrade https://github.com/apache/iceberg-rust/pull/1655
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "colinmarc (via GitHub)" <gi...@apache.org>.
colinmarc commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3266964047
It's currently failing to install protoc on the mac builds:
```
Error: API rate limit exceeded for 13.105.117.136. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3266973661
oh yea the fix for rate limiting is to include the token. the token is automatically set up in CI
```
- name: Install Protoc
uses: arduino/setup-protoc@v3
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3266944270
#1655 is merged, should unblock this. @colinmarc could you try rebasing main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3297437919
> ```
> Error: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
> ```
+1, my suggestion should not be a blocker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3275290217
I think the current error message from `substrait` is pretty informative already
```
Error: Custom { kind: NotFound, error: "Could not find `protoc`. If `protoc` is installed, try setting the `PROTOC` environment variable to the path of the `protoc` binary. To install it on Debian, run `apt-get install protobuf-compiler`. It is also available at https://github.com/protocolbuffers/protobuf/releases For more information: https://docs.rs/prost-build/#sourcing-protoc" }
```
I think we can merge this PR for now, and follow up with another PR to add `protoc` check in the `Makefile` if necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "colinmarc (via GitHub)" <gi...@apache.org>.
colinmarc commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3267003082
Hm, among other things, the MSRV check is still failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3275292345
Thank you for the PR @colinmarc and thanks for the review @liurenjie1024 @fo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3267009986
> Hm, among other things, the MSRV check is still failing.
perhaps an issue related to https://github.com/apache/iceberg-rust/pull/1656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3272936503
> we can potentially do something like this
>
> https://github.com/guacsec/guac/blob/3aa915a571f2c3a097aa236144eb990b46a7fbb6/Makefile#L275-L281
Yes. Installing protobuf is too aggressive. Adding an error message would be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3083435853
> Famous last words! The CI issues seem to mostly revolve around everything needing `protoc` to build now (because of datafusion-substrait).
Is there an issue for this? I don't know on the current status or the reason we're blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "colinmarc (via GitHub)" <gi...@apache.org>.
colinmarc commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3063600973
Famous last words! The CI issues seem to mostly revolve around everything needing `protoc` to build now (because of datafusion-substrait).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "colinmarc (via GitHub)" <gi...@apache.org>.
colinmarc commented on PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501#issuecomment-3273749951
> Adding an error message would be enough.
There's already an error message that I think is pretty clear, and enforcing protoc on our side seems brittle (substrait could remove the requirement in a future release, for instance). What's the reason to add another one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: bump datafusion to 48 [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #1501:
URL: https://github.com/apache/iceberg-rust/pull/1501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org