You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "qrilka (via GitHub)" <gi...@apache.org> on 2023/10/19 07:18:05 UTC
[I] Inconsistent formatting in errors for config options [arrow-datafusion]
qrilka opened a new issue, #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867
### Describe the bug
I was trying to set parquet writer option to 2.0 in datafusion-cli and got the following confusing error message:
```
❯ copy src_table to 'table.parquet' (format parquet,writer_version '2.0');
Invalid or Unsupported Configuration: Unknown or unsupported parquet writer version '2.0' valid options are '1.0' and '2.0'
```
It appears that I could use `2.0` without quotes and with double quotes like `"2.0"` but not with single quotes as the error message reports the values.
Probably the error message should use double quotes instead.
### To Reproduce
Use incorrect value for `writer_version` option in `COPY TO` using parquet format
### Expected behavior
Error message should not be confusing
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770923220
> @devinjdangelo thanks, and what is the proposal then? Not use quotes when they are nor required and single quotes otherwise?
Yes, that is what I would propose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770888028
Perhaps @devinjdangelo has some thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "qrilka (via GitHub)" <gi...@apache.org>.
qrilka commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770919406
@devinjdangelo thanks, and what is the proposal then? Not use quotes when they are nor required and single quotes otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770912986
You can specify writer_version and in fact _almost_ all options without quotes (see below). There are some special cases where datafusion's options parser relies on special characters that `sqlparser-rs` will complain about if it is not wrapped in single quotes. Notably compression options which contain '(' and ')' as well as column specific options using '::'. We could potentially make the UX in those cases nicer by modifying sqlparser-rs to accept those special syntax.
I think cleaning up the error messages and documentation to make all of this more clear is a good idea!
```sql
# Copy parquet with all supported statment overrides
query IT
COPY source_table
TO 'test_files/scratch/copy/table_with_options'
(format parquet,
single_file_output false,
compression snappy,
'compression::col1' 'zstd(5)',
'compression::col2' snappy,
max_row_group_size 12345,
data_pagesize_limit 1234,
write_batch_size 1234,
writer_version 2.0,
dictionary_page_size_limit 123,
created_by 'DF copy.slt',
column_index_truncate_length 123,
data_page_row_count_limit 1234,
bloom_filter_enabled true,
'bloom_filter_enabled::col1' false,
'bloom_filter_fpp::col2' 0.456,
'bloom_filter_ndv::col2' 456,
encoding plain,
'encoding::col1' DELTA_BINARY_PACKED,
'dictionary_enabled::col2' true,
dictionary_enabled false,
statistics_enabled page,
'statistics_enabled::col2' none,
max_statistics_size 123,
bloom_filter_fpp 0.001,
bloom_filter_ndv 100
)
----
2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed issue #7867: Inconsistent formatting in errors for config options
URL: https://github.com/apache/arrow-datafusion/issues/7867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "qrilka (via GitHub)" <gi...@apache.org>.
qrilka commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770208024
If switching from single quotes to double quotes is accepted as a fix I could create a PR doing that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Inconsistent formatting in errors for config options [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7867:
URL: https://github.com/apache/arrow-datafusion/issues/7867#issuecomment-1770887545
Maybe there should simply be no quotes for values that are supposed to be numeric?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org