You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "itsjunetime (via GitHub)" <gi...@apache.org> on 2024/09/26 06:37:35 UTC
[PR] Compare schema as logically equivalent to workaround disappearing metadata [datafusion]
itsjunetime opened a new pull request, #12631:
URL: https://github.com/apache/datafusion/pull/12631
## Which issue does this PR close?
This works around some problems noted in #12560. It is not a full fix, but just unblocks some other work for me.
## Rationale for this change
This check that is being done (which I am changing) only exists to make sure the schema are logically equivalent - if they are different in the metadata of the fields that they contain, that doesn't affect how they are used later on. So it's more 'correct' to just compare them logically as opposed to fully. Comparing them fully does have the benefit of catching issues with lossy transformations between schema (which is probably what is happening in my case to lose the metadata that I mentioned in #12560), but I think that's a benefit that we can get again once we've figured out where metadata is disappearing.
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Compare schema as logically equivalent to workaround disappearing metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12631:
URL: https://github.com/apache/datafusion/pull/12631#issuecomment-2508785091
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Compare schema as logically equivalent to workaround disappearing metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12631: Compare schema as logically equivalent to workaround disappearing metadata
URL: https://github.com/apache/datafusion/pull/12631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Compare schema as logically equivalent to workaround disappearing metadata [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12631:
URL: https://github.com/apache/datafusion/pull/12631#issuecomment-2376350199
I have some suggestion
1. It would be nice if there is a reproducible example so we could find out why and where metadata loss. It could also avoid breaking your case again if we added the metadata check back again without fixing the real issue.
2. If the reproducible example is hard to find, we could avoid the metadata check but other than `logically_equivalent_names_and_types` we need to check `nullable` as well. Try modify the `logically_equivalent_names_and_types` or create another new function that checks `nullable`. Also, add a TODO aside it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Compare schema as logically equivalent to workaround disappearing metadata [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #12631:
URL: https://github.com/apache/datafusion/pull/12631#issuecomment-2378161907
> I think this is a reasonable change, but it does, as you say, potentially cause silent errors rather than explicit ones.
>
> This is a good idea for Influx, but maybe not as good an idea for other users of DataFusion
>
> Something we have done in the past for situations like this was to have a [config flag](https://datafusion.apache.org/user-guide/configs.html) that could control the behavior
>
> For example, this flag
>
> ```
> datafusion.optimizer.skip_failed_rules
> ```
>
> Controls if a failure in an optimizer rule would reject the entire plan or just skip that particular failed rule
>
> Perhaps we could make a similar flag for schema checks like
>
> ```
> datafusion.optimizer.only_compare_logical_types
> ```
>
> Or something 🤔
Does `only_compare_logical_types` includes `nullable`? If yes, it looks good to me. If not should we need another `only_compare_logical_types_with_nullable` flag as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org