You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "logan-keede (via GitHub)" <gi...@apache.org> on 2025/03/22 16:38:09 UTC
[I] [DISCUSS] In-house implementation of certain dependency package. [datafusion]
logan-keede opened a new issue, #15360:
URL: https://github.com/apache/datafusion/issues/15360
### Is your feature request related to a problem or challenge?
DataFusion has many dependencies that leads to increased binary size and compilation while saving us the trouble of maintaining those implementations in our own codebase.
### Describe the solution you'd like
This issue is to Identify any crate/dependency `shallow` and `stable` enough to justify having an In-house implementation.
'shallow': We do not use much of the original crate. i.e we just use a small part that can be implemented in house without having to implement/port a massive dependency tree that crate has. The shallower the dependency, the more beneficial an in-house implementation would be.
'stable': The code is mostly stable and will not require much active changes after having an In-house implementation. (While all code eventually requires some changes, it is about how frequently and disruptively it will give us a headache.)
### Describe alternatives you've considered
Do not have any in-house implementation. Pay the price of increased binary size and compile time.
### Additional context
_Originally suggested by @ozankabak while discussing my GSoC 2025 Proposal for "Optimizing compile time and binary size" over discord and before that in #14478._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Consider Vendoring Certain Dependencies [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2758955432
Right, but we spend a lot of time on CI and debugging without LTO. Also, dependency creep increases the number of "moving" parts and makes maintenance harder. Therefore I think this is a worthwhile exercise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Consider Vendoring Certain Dependencies [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2758703831
Pretty sure the linker/LLVM filters out unused code anyways- at least if LTO is on which it is for release builds. It might help decrease build time marginally but I really wouldn't expect the actual executable size to be significantly different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Consider Vendoring Certain Dependencies [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2759145877
I think that since rust effectively statically links all binaries vendoring dependencies will not likely make any difference in binary size
The only benefit of vendoring stuff would be the dependecy creep issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Consider Vendoring Certain Dependencies [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2759173968
IMO dependency creep is the most important issue here. For the others, we can run experiments to see what happens to compilation times and debug mode binary sizes. I don't think there is much point in engaging theoretical discussions on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] In-house implementation of certain dependencies. [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2745373075
If we can write a script to approximately estimate "how much" of an external crate we are using, we can create a starter list of candidates for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] In-house implementation of certain dependencies. [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2747664199
FWIW I think copying parts of third-party crates is often called "vendoring the dependencies" . Using that term might help communicate what this is proposing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [DISCUSS] Consider Vendoring Certain Dependencies [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #15360:
URL: https://github.com/apache/datafusion/issues/15360#issuecomment-2759069227
I can't say I agree with you there but I'm only one voice. I'd rather see the effort put into other areas tbh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org