You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "liurenjie1024 (via GitHub)" <gi...@apache.org> on 2025/04/07 13:06:04 UTC
[PR] Change tokio feature [iceberg-rust]
liurenjie1024 opened a new pull request, #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173
## What changes are included in this PR?
Changes tokio dependency, enable `sync` feature only by default, but requires `full` feature when we enable `tokio` cfg. With this change, we can still use tokio's runtime indepdenent utils, while only rely on tokio's runtime when we enable `tokio` cfg.
## Are these changes tested?
CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785035590
I suggest we wait a bit to see how https://github.com/apache/iceberg-rust/issues/1036 progresses. Ideally, all dependencies on `cfg(tokio)` can be eliminated once our mini engine has been extracted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2786265611
This is the dependency tree of tokio:
`cargo tree -i tokio -p iceberg -e features`
```
tokio v1.44.2
└── hyper-util v0.1.10
├── hyper-util feature "client"
│ └── reqwest v0.12.12
│ └── reqsign v0.16.1
│ └── opendal v0.51.2
│ ├── opendal feature "default"
│ │ └── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg)
│ │ ├── iceberg feature "default" (command-line)
│ │ │ └── iceberg-catalog-memory v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/catalog/memory)
│ │ │ └── iceberg-catalog-memory feature "default"
│ │ │ [dev-dependencies]
│ │ │ └── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg) (*)
│ │ ├── iceberg feature "storage-fs"
│ │ │ └── iceberg feature "default" (command-line) (*)
│ │ ├── iceberg feature "storage-memory"
│ │ │ └── iceberg feature "default" (command-line) (*)
│ │ ├── iceberg feature "storage-s3"
│ │ │ └── iceberg feature "default" (command-line) (*)
│ │ └── iceberg feature "tokio"
│ │ └── iceberg feature "default" (command-line) (*)
│ ├── opendal feature "executors-tokio"
│ │ └── opendal feature "default" (*)
│ ├── opendal feature "internal-tokio-rt"
│ │ └── opendal feature "services-fs"
│ │ └── iceberg feature "storage-fs" (*)
│ ├── opendal feature "services-fs" (*)
│ ├── opendal feature "services-memory"
│ │ ├── iceberg feature "storage-memory" (*)
│ │ └── opendal feature "default" (*)
│ └── opendal feature "services-s3"
│ └── iceberg feature "storage-s3" (*)
│ ├── reqsign feature "reqwest_request"
│ │ └── opendal feature "services-s3" (*)
│ └── reqsign feature "services-aws"
│ └── opendal feature "services-s3" (*)
│ ├── reqwest feature "__rustls"
│ │ └── reqwest feature "rustls-tls-webpki-roots-no-provider"
│ │ └── reqwest feature "rustls-tls-webpki-roots"
│ │ └── reqwest feature "rustls-tls"
│ │ └── opendal feature "default" (*)
│ ├── reqwest feature "__rustls-ring"
│ │ └── reqwest feature "rustls-tls-webpki-roots" (*)
│ ├── reqwest feature "__tls"
│ │ └── reqwest feature "__rustls" (*)
│ ├── reqwest feature "json"
│ │ └── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg) (*)
│ ├── reqwest feature "rustls-tls" (*)
│ ├── reqwest feature "rustls-tls-webpki-roots" (*)
│ ├── reqwest feature "rustls-tls-webpki-roots-no-provider" (*)
│ └── reqwest feature "stream"
│ └── opendal v0.51.2 (*)
│ └── hyper-util feature "client-legacy"
│ ├── hyper-rustls v0.27.5
│ │ ├── hyper-rustls feature "http1"
│ │ │ └── reqwest v0.12.12 (*)
│ │ ├── hyper-rustls feature "ring"
│ │ │ └── reqwest feature "__rustls-ring" (*)
│ │ ├── hyper-rustls feature "tls12"
│ │ │ └── reqwest v0.12.12 (*)
│ │ ├── hyper-rustls feature "webpki-roots"
│ │ │ └── hyper-rustls feature "webpki-tokio"
│ │ │ └── reqwest feature "rustls-tls-webpki-roots-no-provider" (*)
│ │ └── hyper-rustls feature "webpki-tokio" (*)
│ └── reqwest v0.12.12 (*)
├── hyper-util feature "client-legacy" (*)
├── hyper-util feature "default"
│ └── reqwest v0.12.12 (*)
├── hyper-util feature "http1"
│ └── reqwest v0.12.12 (*)
│ └── hyper-rustls feature "http1" (*)
└── hyper-util feature "tokio"
├── hyper-rustls v0.27.5 (*)
└── reqwest v0.12.12 (*)
├── hyper-util feature "client-legacy" (*)
└── hyper-util feature "tokio" (*)
├── tokio feature "bytes"
│ └── tokio feature "io-util"
│ ├── opendal v0.51.2 (*)
│ └── parquet v54.2.0
│ ├── parquet feature "arrow"
│ │ └── parquet feature "default"
│ │ └── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg) (*)
│ ├── parquet feature "arrow-array"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-buffer"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-cast"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-data"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-ipc"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-schema"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "arrow-select"
│ │ └── parquet feature "arrow" (*)
│ ├── parquet feature "async"
│ │ └── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg) (*)
│ ├── parquet feature "base64"
│ │ ├── parquet feature "arrow" (*)
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "brotli"
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "default" (*)
│ ├── parquet feature "flate2"
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "futures"
│ │ └── parquet feature "async" (*)
│ ├── parquet feature "lz4"
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "lz4_flex"
│ │ └── parquet feature "lz4" (*)
│ ├── parquet feature "simdutf8"
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "snap"
│ │ └── parquet feature "default" (*)
│ ├── parquet feature "tokio"
│ │ └── parquet feature "async" (*)
│ ├── parquet feature "zstd"
│ │ └── parquet feature "default" (*)
│ └── parquet feature "zstd-sys"
│ └── parquet feature "zstd" (*)
│ └── reqwest feature "__tls" (*)
├── tokio feature "default"
│ ├── backon v1.3.0
│ │ ├── backon feature "default"
│ │ │ └── opendal v0.51.2 (*)
│ │ ├── backon feature "gloo-timers-sleep"
│ │ │ └── backon feature "default" (*)
│ │ ├── backon feature "std-blocking-sleep"
│ │ │ └── backon feature "default" (*)
│ │ └── backon feature "tokio-sleep"
│ │ └── opendal v0.51.2 (*)
│ │ └── backon feature "default" (*)
│ ├── hyper v1.6.0
│ │ └── hyper-rustls v0.27.5 (*)
│ │ ├── hyper feature "client"
│ │ │ └── reqwest v0.12.12 (*)
│ │ │ └── hyper-util feature "client" (*)
│ │ ├── hyper feature "default"
│ │ │ ├── hyper-util v0.1.10 (*)
│ │ │ └── reqwest v0.12.12 (*)
│ │ └── hyper feature "http1"
│ │ └── reqwest v0.12.12 (*)
│ │ └── hyper-util feature "http1" (*)
│ ├── hyper-rustls v0.27.5 (*)
│ ├── opendal v0.51.2 (*)
│ ├── tokio-rustls v0.26.1
│ │ └── hyper-rustls v0.27.5 (*)
│ │ ├── tokio-rustls feature "ring"
│ │ │ └── reqwest feature "__rustls-ring" (*)
│ │ └── tokio-rustls feature "tls12"
│ │ └── reqwest v0.12.12 (*)
│ │ └── hyper-rustls feature "tls12" (*)
│ ├── tokio-util v0.7.13
│ │ ├── tokio-util feature "codec"
│ │ │ └── reqwest v0.12.12 (*)
│ │ └── tokio-util feature "io"
│ │ └── reqwest v0.12.12 (*)
│ └── tower v0.5.2
│ ├── tower feature "__common"
│ │ └── tower feature "util"
│ │ └── reqwest v0.12.12 (*)
│ ├── tower feature "futures-core"
│ │ └── tower feature "__common" (*)
│ ├── tower feature "futures-util"
│ │ └── tower feature "util" (*)
│ ├── tower feature "pin-project-lite"
│ │ ├── tower feature "__common" (*)
│ │ ├── tower feature "timeout"
│ │ │ └── reqwest v0.12.12 (*)
│ │ └── tower feature "util" (*)
│ ├── tower feature "sync_wrapper"
│ │ └── tower feature "util" (*)
│ ├── tower feature "timeout" (*)
│ ├── tower feature "tokio"
│ │ └── tower feature "timeout" (*)
│ └── tower feature "util" (*)
├── tokio feature "fs"
│ ├── opendal feature "services-fs" (*)
│ └── reqwest feature "stream" (*)
├── tokio feature "io-util" (*)
├── tokio feature "libc"
│ └── tokio feature "net"
│ └── reqwest v0.12.12 (*)
│ └── hyper-util feature "tokio" (*)
├── tokio feature "macros"
│ └── parquet v54.2.0 (*)
├── tokio feature "mio"
│ └── tokio feature "net" (*)
├── tokio feature "net" (*)
├── tokio feature "rt"
│ └── parquet v54.2.0 (*)
│ ├── opendal feature "executors-tokio" (*)
│ ├── tokio feature "rt-multi-thread"
│ │ └── opendal feature "internal-tokio-rt" (*)
│ └── hyper-util feature "tokio" (*)
├── tokio feature "rt-multi-thread" (*)
├── tokio feature "socket2"
│ └── tokio feature "net" (*)
├── tokio feature "sync"
│ ├── hyper v1.6.0 (*)
│ ├── iceberg v0.4.0 (/Users/xxchan/Projects/iceberg-rust/crates/iceberg) (*)
│ ├── opendal v0.51.2 (*)
│ ├── tokio-util v0.7.13 (*)
│ └── tower v0.5.2 (*)
│ └── hyper-util feature "client-legacy" (*)
├── tokio feature "time"
│ └── reqwest v0.12.12 (*)
│ ├── backon feature "tokio-sleep" (*)
│ ├── hyper-util feature "tokio" (*)
│ └── tower feature "timeout" (*)
└── tokio feature "tokio-macros"
└── tokio feature "macros" (*)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2786317414
> To summarize, currently we already enable tokio runtime features unconditionally, even when feature `tokio` is not enabled, since other dependencies enabled it.
Actually `tokio` is a hard requirement for opendal now? 🤔 Since `executors-tokio` is the only available executor now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785051178
> I suggest we wait a bit to see how #1036 progresses. Ideally, all dependencies on `cfg(tokio)` can be eliminated once our mini engine has been extracted.
@Xuanwo Seems we only need to enable `rt-multithread`, WDYT? This is blocking #1165 , so I want to have it merged first. This pr also aligns with the goal of removing unnecessary `cfg(tokio)` , more details could be found https://github.com/apache/iceberg-rust/pull/1165#discussion_r2026865964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2788141141
> To summarize, currently we already enable tokio runtime features unconditionally, even when feature `tokio` is not enabled, since other dependencies enabled it.
>
> Therefore, I think the best way to proceed is:
>
> * In this PR, we just need to remove the "optional" for `tokio` to unblock [feat(puffin): Make Puffin APIs public #1165](https://github.com/apache/iceberg-rust/pull/1165).
> * In a follow-up PR, we can gradually turn off unnecessary features of dependencies.
I agree that we should eventually remove unnecessary tokio runtime dependency, but I don't think we should revert the change of `tokio/rt-multi-thread`, as this is the goal we want to achieve. Thinking about the final status, it should be what looks like in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2784829700
@Xuanwo for context, I believe [this thread](https://github.com/apache/iceberg-rust/pull/1165#discussion_r2026865964) is at least part of the reason why @liurenjie1024 is proposing this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2783462518
`tokio/full` is really big. Do we really want it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "xxchan (via GitHub)" <gi...@apache.org>.
xxchan commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2786279257
To summarize, currently we already enable tokio runtime features unconditionally, even when feature `tokio` is not enabled, since other dependencies enabled it.
Therefore, I think the best way to proceed is:
- In this PR, we just need to remove the "optional" for `tokio`.
- In a follow-up PR, we can gradually turn off unnecessary features of dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Change tokio feature [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1173:
URL: https://github.com/apache/iceberg-rust/pull/1173#issuecomment-2785060418
> WDYT if I change full to `rt-multithread`
Makes more sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org