You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/12/04 17:52:36 UTC
[I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
alamb opened a new issue, #13648:
URL: https://github.com/apache/datafusion/issues/13648
### Is your feature request related to a problem or challenge?
This is broken out from a more general ticket here
- https://github.com/apache/datafusion/issues/13525
🥳 In my opinion DataFusion is now good enough (performance and feature wise) for many people to have buit real systems and products
However, as more people build "real" systems using DataFusion, our historic "move fast and break things and hope you can keep up" mentality likely needs to adjust to a more mature "move as fast as possible, but minimize breakages" type response.
My summary of the discussion on https://github.com/apache/datafusion/issues/13525 from @findepi @scsmithr @waynexia @timsaucer @Rachelint @Omega359 @jonmmease @Dandandan and @andygrove was that many existing heavy users of DataFusion spend a lot of time during upgrades from one DataFusion release to another
Specifically, I think the core challenge I heard was *NOT* the mechnical API changes required, but the effort required to diagnose more suble issues such as:
- Queries that used to work stopped working
### Describe the solution you'd like
I would like to improve the ease of upgrading DataFusion versions
There are many ways to do so and I would like to use this ticket to capture / organize the work in this area
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "jonmmease (via GitHub)" <gi...@apache.org>.
jonmmease commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-3019224293
Thanks, no I hadn't come across these, and the look great!. My habit was just to search for the [CHANGELOG](https://github.com/apache/datafusion/blob/main/dev/changelog/48.0.0.md) when upgrading. I'll definitely review these for the next upgrade, and they certainly look like great LLM context as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2520842461
Here is a cockroachDB article that might be found interesting on fuzz testing the DB with generating random(valid/not valid) statements through SQLSmith
https://www.cockroachlabs.com/blog/sqlsmith-randomized-sql-testing/
https://www.cockroachlabs.com/blog/testing-random-valid-sql-in-cockroachdb/
I understand SQLancer is the way, but we may consider this as an alternative just in case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-3018739845
I added a ticket to discuss less frequent major releases
- https://github.com/apache/datafusion/issues/16622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2557109771
I found a good previous discussion by @andygrove about potential patch releases:
- https://github.com/apache/datafusion/issues/5269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "crepererum (via GitHub)" <gi...@apache.org>.
crepererum commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2520616605
I would like to propose #13665 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518166067
> So before release, we should list out the possible changes that would need to be made if a upgrade were to happen during the development process.
I think it is a great idea. The challenge will be identifying such changes I thunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "jonmmease (via GitHub)" <gi...@apache.org>.
jonmmease commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-3019095705
As I've come to use Claude Code more for development, I've found that it's pretty good at performing version updates for Rust crates using Release Note + Compiler Errors + Tests. If each breaking change were required to have migration instructions, with examples, in the release notes (or a separate migration guide), I think the upgrade process could be pretty reliable and automated with LLM assistance. And of course this would be useful for the humans as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-6365345951
I think this one has served its usefulness and most items have been completed so closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518161520
Yes, I think one of the comments in that discussion mentioned that certain changes that would cause breakage should be mentioned in every release. So before release, we should list out the possible changes that would need to be made if a upgrade were to happen during the development process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518254795
There is an interesting approach at MariaDB, they generate queries with different syntaxes to find regressions.
Basically we can take their main.sql file which is 7MB of different queries including join queries and adapt it to DF.
There is no answers check, just smoke test that query can run successfully
The example can be found https://github.com/mariadb-corporation/mariadb-qa/tree/master/pquery
@alamb WDYT? it looks like a low hanging fruit, we can take the file and run it in latest datafusion CLI as part of CI or major release verification process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-3019110593
That is a great idea @jonmmease -- I wonder if you have seen the new upgrade notes for the last few releases
- https://datafusion.apache.org/library-user-guide/upgrading.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13648: [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade)
URL: https://github.com/apache/datafusion/issues/13648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-3019252931
We should probably update the changelog generator script to include a link to the upgrade guide 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518293018
Thats frustrating. Lets see if sqlancer can generate something similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518265023
> There is an interesting approach at MariaDB, they generate queries with different syntaxes to find regressions. Basically we can take their main.sql file which is 7MB of different queries including join queries and adapt it to DF.
>
> There is no answers check, just smoke test that query can run successfully
>
> The example can be found https://github.com/mariadb-corporation/mariadb-qa/tree/master/pquery
>
> @alamb WDYT? it looks like a low hanging fruit, we can take the file and run it in latest datafusion CLI as part of CI or major release verification process
Almost absolutely NOT. https://github.com/mariadb-corporation/mariadb-qa/blob/master/LICENSE.md
https://www.apache.org/legal/resolved.html#category-x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] A collection of items to improve DataFuson stability (reduce effort required to upgrade) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13648:
URL: https://github.com/apache/datafusion/issues/13648#issuecomment-2518258647
> @alamb WDYT? it looks like a low hanging fruit, we can take the file and run it in latest datafusion CLI as part of CI or major release verification process
I think in general the more testing we have the better. This idea sounds good to me -- I think more fully leveraging @2010YOUY01 's integration into sqlancer is also quite interesting.
Let's try and write some tickets to capture these ideas too - I can spend some time working on this over the next day or two
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org