You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/12/17 15:06:15 UTC
[I] Improved developer / CI experience [datafusion]
alamb opened a new issue, #13813:
URL: https://github.com/apache/datafusion/issues/13813
### Is your feature request related to a problem or challenge?
As DataFusion
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2549401931
Is there a way to test changes to the ci pipeline without actually checking in files to DF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2579414080
good find!
> 2024 edition though...
but maybe it's not a bad thing? https://doc.rust-lang.org/edition-guide/editions/#editions-do-not-split-the-ecosystem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-3202037457
For transparency: we want to try the merge queue https://github.com/apache/datafusion/issues/6880
> I also pinged the infra team to see if we can get larger runners
The infra team said they are not available now. I also pinged GitHub - I believe there's a discussion about providing more powerful runners, but no decisions have been made yet.
I also chatted with the Apache Fundraising team - it's possible to submit a "need" we have (=compute for large runners), so if a company is ever willing, ASF will match us. Happy to manage the process on our side if there any no objections?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2549410610
> Is there a way to test changes to the ci pipeline without actually checking in files to DF?
We could in theory change the triggering rules for the the jobs to run on any changes to `.github` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2585836320
> good find!
>
> > 2024 edition though...
>
> but maybe it's not a bad thing? https://doc.rust-lang.org/edition-guide/editions/#editions-do-not-split-the-ecosystem
I tried to compile DF under nightly with 2024 edition ... there are quite a few code changes that will have to be made. #1 is rng.gen => rng.r#gen (gen is now a reserved keyword). #2 is
```
error: this pattern relies on behavior which may change in edition 2024
--> datafusion/proto-common/src/from_proto/mod.rs:443:49
|
443 | DataType::Dictionary(_, ref value_type) => {
| ^^^ cannot override to bind by-reference when that is the implicit default
|
= note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2024/match-ergonomics.html>
help: make the implied reference pattern explicit
|
443 | &DataType::Dictionary(_, ref value_type) => {
| +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2556243727
I've managed to knock some time off of the ci runs on a sidebranch (need to rebase off of main before pushing a PR) bu here is what I've managed to get to so far:
an example of a PR run now:
my sidebranch on the second run (cache populated)
cargo test doc and hash collisions are the two ones right now that take the most time and much of that is DF compilation that can't really be cached (would blow out the 10GB cache limit I think).
Most of the improvement is better caching, splitting up the linux build test into multiple jobs (it's all cargo checks), and switching to nextest for many tests. I also disabled the intel mac build test.
I hope to submit a PR this weekend for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-3064077568
Added cache to CI runners to get some speedup: https://github.com/apache/datafusion/pull/16709
Also pinged the infra team if we can get larger runners: https://issues.apache.org/jira/browse/INFRA-27000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2550544724
> Is there a way to test changes to the ci pipeline without actually checking in files to DF?
@Omega359 you can enable the workflow(s) on your fork
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2549419895
If no one beats me to this I can likely take a look at the ci pipeline during the holiday break. I still want to finish the sqlite test integration first if possible though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2549627397
I also hope to hack on various parts of this EPIC during thebreak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2557117029
> I've managed to knock some time off of the ci runs on a sidebranch (need to rebase off of main before pushing a PR) but here is what I've managed to get to so far:
LOL your screenshots show 37m --> 19m (which is a 2x improvement by my accounting) so that is a somewhat modest description :)
> cargo test doc and hash collisions are the two ones right now that take the most time and much
I think the hash collisions one fails so irregularly, that we should consider simply running it on main (not on all PRs) as discussed in https://github.com/apache/datafusion/issues/13845. While that does indeed defer the potential for finding issues I think the tradeoff might be worth it in this specific case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Alexhuszagh (via GitHub)" <gi...@apache.org>.
Alexhuszagh commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2549040560
If this could be improved it would be wonderful. Arrow and Datafusion are very important projects using lexical and I want to test all my code against them and other high impact projects as an additional precaution in addition to our own checks internally, so being able to run these tests locally would be a huge benefit on my end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve developer / CI speed [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13813:
URL: https://github.com/apache/datafusion/issues/13813#issuecomment-2578956224
FYI, I came across [this today](https://github.com/rust-lang/rust/pull/126245) with a great example of how much faster doctests can be in [this comment](https://github.com/rust-lang/rust/pull/126245#issuecomment-2291429124). 2024 edition though...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org