You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tinfoil-knight (via GitHub)" <gi...@apache.org> on 2024/03/13 10:44:31 UTC
[PR] support unprefixed config format options [arrow-datafusion]
tinfoil-knight opened a new pull request, #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #9575
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
#9382 standardised write configuration but broke backwards compatibility for the format specific options of the `COPY` command.
Earlier users could have a query like this without prefixing all options:
```
COPY source_table TO 'test_files/scratch/copy/format_table/' (
format parquet,
compression snappy,
'compression::col1' 'zstd(5)'
);
```
But after #9382, they'd need to prefix every format-specific options in this verbose way:
```
COPY source_table TO 'test_files/scratch/copy/format_table/' (
format parquet,
'parquet.compression' snappy,
'parquet.compression::col1' 'zstd(5)'
);
```
This PR adds back the support for format-specific options that worked without providing the format prefix.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
This PR renames keys without the file_type/format prefix for file types supported by the `COPY` command for backwards compatibility.
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
- Existing tests in `copy.slt` confirm that the prefixed format options (eg. `(format json, 'json.compression' gzip)`) added in #9382 work & aren't broken.
- Tests have been added for the legacy format (eg. `(format json, compression gzip)`).
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
Yes. But the current documentation (https://arrow.apache.org/datafusion/user-guide/sql/write_options.html#available-options) lists the legacy format so this PR doesn't require a doc update.
> If there are any breaking changes to public APIs, please add the `api change` label.
This PR restores breaking changes so I'm not adding the `api change` label. Please let me know if you think we should add the label nonetheless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-1995225783
Thank you @tinfoil-knight for this. We will do a quick follow-up finalizing the design in #9382. Let's get the follow-up in and then modify this/get this in afterwards for backwards compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2007709217
I think we should also consider holding the release for this PR (see #9682 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on code in PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#discussion_r1523217543
##########
datafusion/sql/src/statement.rs:
##########
@@ -855,6 +855,26 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
let file_type = try_infer_file_type(&mut options, &statement.target)?;
let partition_by = take_partition_by(&mut options);
+ match &file_type {
+ // Renames un-prefixed keys to support legacy format specific options
+ FileType::CSV | FileType::JSON | FileType::PARQUET => {
+ let prefix = format!("{}", file_type);
+ let keys_to_rename: Vec<_> = options
+ .keys()
+ .filter(|key| !key.starts_with(&prefix))
+ .cloned()
+ .collect();
+
+ for key in keys_to_rename {
+ if let Some(value) = options.remove(&key) {
+ let new_key = format!("{}.{}", prefix, key);
+ options.insert(new_key, value);
+ }
+ }
+ }
+ _ => {}
+ }
+
Review Comment:
The overall strategy here looks good, but I propose two changes:
1. We can make the logic more concise by modifying options in a single pass with into_iter, map, collect. This also avoids clones.
2. I think we should be a bit more conservative about the keys we modify. If the key has a namespace at all (rather than specifically the file format namespace), we can leave it as is. Downstream users may for example add a custom namespace and we wouldn't wan't this code to modify it unexpectedly.
##########
datafusion/sql/src/statement.rs:
##########
@@ -855,6 +855,26 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
let file_type = try_infer_file_type(&mut options, &statement.target)?;
let partition_by = take_partition_by(&mut options);
+ match &file_type {
+ // Renames un-prefixed keys to support legacy format specific options
+ FileType::CSV | FileType::JSON | FileType::PARQUET => {
+ let prefix = format!("{}", file_type);
+ let keys_to_rename: Vec<_> = options
+ .keys()
+ .filter(|key| !key.starts_with(&prefix))
+ .cloned()
+ .collect();
+
+ for key in keys_to_rename {
+ if let Some(value) = options.remove(&key) {
+ let new_key = format!("{}.{}", prefix, key);
+ options.insert(new_key, value);
+ }
+ }
+ }
+ _ => {}
+ }
+
Review Comment:
```suggestion
let options = match &file_type {
// Renames un-prefixed keys to support legacy format specific options
FileType::CSV | FileType::JSON | FileType::PARQUET => {
options
.into_iter()
.map(|(k, v)| {
// If config does not belong to any namespace, assume it is
// a legacy option and apply file_type namespace for backwards
// compatibility.
if !k.contains('.') {
let new_key = format!("{}.{}", file_type, k);
(new_key, v)
} else {
(k, v)
}
})
.collect()
}
_ => options,
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#discussion_r1523483285
##########
datafusion/sqllogictest/test_files/copy.slt:
##########
@@ -484,3 +484,35 @@ COPY (select col2, sum(col1) from source_table
# Copy from table with non literal
query error DataFusion error: SQL error: ParserError\("Expected ',' or '\)' after option definition, found: \+"\)
COPY source_table to '/tmp/table.parquet' (row_group_size 55 + 102);
+
+
+# Legacy Format Options Support
Review Comment:
❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on code in PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#discussion_r1523729912
##########
datafusion/sql/src/statement.rs:
##########
@@ -855,6 +855,31 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
let file_type = try_infer_file_type(&mut options, &statement.target)?;
let partition_by = take_partition_by(&mut options);
+ let rename_keys = |options: HashMap<String, String>| {
+ options
+ .into_iter()
+ .map(|(k, v)| {
+ // If config does not belong to any namespace, assume it is
+ // a legacy option and apply file_type namespace for backwards
+ // compatibility.
+ if !k.contains('.') {
+ let new_key = format!("{}.{}", file_type, k);
+ (new_key, v)
+ } else {
+ (k, v)
+ }
+ })
+ .collect()
+ };
+
+ // Renames un-prefixed keys to support legacy format specific options
+ let options = match &file_type {
+ FileType::CSV | FileType::JSON => rename_keys(options),
+ #[cfg(feature = "parquet")]
+ FileType::PARQUET => rename_keys(options),
Review Comment:
@devinjdangelo
Since `FileType::PARQUET` isn't accessible without the feature enabled, I added the `cfg` macro but I'm getting:
`code is inactive due to #[cfg] directives: feature = "parquet" is disabled` from rust-analyzer.
I haven't been able to figure out why. I checked all other instances of `#[cfg(feature = "parquet")]` but didn't get the disabled warning anywhere else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on code in PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#discussion_r1523723195
##########
datafusion/sql/src/statement.rs:
##########
@@ -855,6 +855,26 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
let file_type = try_infer_file_type(&mut options, &statement.target)?;
let partition_by = take_partition_by(&mut options);
+ match &file_type {
+ // Renames un-prefixed keys to support legacy format specific options
+ FileType::CSV | FileType::JSON | FileType::PARQUET => {
+ let prefix = format!("{}", file_type);
+ let keys_to_rename: Vec<_> = options
+ .keys()
+ .filter(|key| !key.starts_with(&prefix))
+ .cloned()
+ .collect();
+
+ for key in keys_to_rename {
+ if let Some(value) = options.remove(&key) {
+ let new_key = format!("{}.{}", prefix, key);
+ options.insert(new_key, value);
+ }
+ }
+ }
+ _ => {}
+ }
+
Review Comment:
Implemented both changes.
Converted the PR to draft for the while being due to https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-1995225783.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2009814421
@devinjdangelo I'm not sure if the original issue is valid anymore. After #9604, you can use formatting options without prefixing them with the format. See this for an example:
https://github.com/apache/arrow-datafusion/blob/89efc4a7e06bd0295ca72dd6ec5fe987d1ac246b/datafusion/sqllogictest/test_files/copy.slt#L22-L32
I think this PR & the linked issue should be closed.
---
I also found a CLI bug while working on this issue & I've filed: #9707 & pushed a PR for it separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2010429822
Update:
https://github.com/apache/arrow-datafusion/issues/9716 is opened now for the format prefix issue.
@alamb I'm closing this PR since it doesn't seem relevant anymore. See the previous comment (https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2009814421) for more details.
Please close the issue: #9575 now as it's stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-1995254996
> We will do a quick follow-up finalizing the design in #9382. Let's get the follow-up in and then modify this/get this in afterwards for backwards compatibility.
Okay. Sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2007795789
> I think we should also consider holding the release for this PR (see #9682 ) - so if you need help finishing it up @tinfoil-knight please just let us know
I'll fix the tests by tomorrow & will let you know if anything comes up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#issuecomment-2007146368
@tinfoil-knight I think this should be unblocked now that #9382 is closed. If you have a moment to rebase/resolve any conflicts and CI failures, I think we can get this merged after a final review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "tinfoil-knight (via GitHub)" <gi...@apache.org>.
tinfoil-knight closed pull request #9594: support unprefixed config format options
URL: https://github.com/apache/arrow-datafusion/pull/9594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support unprefixed config format options [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on code in PR #9594:
URL: https://github.com/apache/arrow-datafusion/pull/9594#discussion_r1524728358
##########
datafusion/sql/src/statement.rs:
##########
@@ -855,6 +855,31 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
let file_type = try_infer_file_type(&mut options, &statement.target)?;
let partition_by = take_partition_by(&mut options);
+ let rename_keys = |options: HashMap<String, String>| {
+ options
+ .into_iter()
+ .map(|(k, v)| {
+ // If config does not belong to any namespace, assume it is
+ // a legacy option and apply file_type namespace for backwards
+ // compatibility.
+ if !k.contains('.') {
+ let new_key = format!("{}.{}", file_type, k);
+ (new_key, v)
+ } else {
+ (k, v)
+ }
+ })
+ .collect()
+ };
+
+ // Renames un-prefixed keys to support legacy format specific options
+ let options = match &file_type {
+ FileType::CSV | FileType::JSON => rename_keys(options),
+ #[cfg(feature = "parquet")]
+ FileType::PARQUET => rename_keys(options),
Review Comment:
@tinfoil-knight I did not observe this when I pulled down the latest changes locally. In my experience, rust-analyzer is amazing but can also be flaky with large/complex workspaces like datafusion. I occasionally need to do a "hard reset" to get rid of strange errors or warnings like this which I do not believe are correct (i.e. cargo check disagrees with rust-analyzer).
1. Close and reopen VS code
2. If 1 doesn't work, cargo clean and then close/reopen VS code
When working with non default features, it can be helpful to force rust analyzer to enable all features. You can add this to your settings.json for VScode:
```
"rust-analyzer.cargo.features": [
"all"
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org