You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2023/03/24 18:11:26 UTC
[GitHub] [arrow-datafusion] tustvold opened a new issue, #5725: Depend on Arrow Subcrates
tustvold opened a new issue, #5725:
URL: https://github.com/apache/arrow-datafusion/issues/5725
### Is your feature request related to a problem or challenge?
Currently the various DataFusion components all depend on `arrow`, this means that it must wait for the entirety of arrow-rs to compile before it can start compiling crates from datafusion. This is despite crates like datafusion_common likely not needing to depend on heavy crates like arrow-ord and arrow-arith.
### Describe the solution you'd like
I'd like DataFusion crates to depend on the relevant subcrates, instead of the top-level kitchen sink crate
### Describe alternatives you've considered
We could not do this
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Depend on Arrow Subcrates [datafusion]
Posted by "tmi (via GitHub)" <gi...@apache.org>.
tmi commented on issue #5725:
URL: https://github.com/apache/datafusion/issues/5725#issuecomment-2168560427
Honestly I think this issue should be closed as there won't be that much value -- e.g., `datafusion-common` depends on pyarrow which doesn't exist as a standalone crate so you are stuck with `arrow`, as well as depending on at least 10 arrow subcrates, including the mentioned `arrow-ord`, for other reasons...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Depend on Arrow Subcrates [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5725:
URL: https://github.com/apache/datafusion/issues/5725#issuecomment-2359387437
I agree -- the work described in this issue only makes it harder to work with datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] yash-gupta167 commented on issue #5725: Depend on Arrow Subcrates
Posted by "yash-gupta167 (via GitHub)" <gi...@apache.org>.
yash-gupta167 commented on issue #5725:
URL: https://github.com/apache/arrow-datafusion/issues/5725#issuecomment-1484099057
Refactor the DataFusion ```Cargo.toml``` files. For each DataFusion component, update the ```Cargo.toml``` file to replace the top-level Arrow dependency with the specific subcrates identified .
For example, if a DataFusion component currently has the following dependency in its Cargo.toml:
```
[dependencies]
arrow = "x.y.z"
```
And it only needs the ```arrow-array```, ```arrow-ipc```, and ```arrow-compute``` subcrates, you can update the ```Cargo.toml``` file as follows:
```
[dependencies]
arrow-array = { version = "x.y.z", package = "arrow" }
arrow-ipc = { version = "x.y.z", package = "arrow" }
arrow-compute = { version = "x.y.z", package = "arrow" }
```
-Update the DataFusion source code: Modify the import statements in the DataFusion source code to use the specific subcrates instead of the top-level Arrow crate. This may require updating module paths and potentially some minor code changes to accommodate the new dependencies.
-Test the refactored DataFusion components: After refactoring the dependencies, run the test suite to ensure that all DataFusion components still function correctly. Address any issues that arise during testing.
-Benchmark and document the improvements: Measure and document the improvements in compilation time and binary size reduction as a result of this refactoring. This can serve as a reference for future optimizations.
Please Let me know if this solution is relevant or am I going wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Depend on Arrow Subcrates [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5725: Depend on Arrow Subcrates
URL: https://github.com/apache/datafusion/issues/5725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org