You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/06/19 17:52:11 UTC
[I] Make `datafusion-cli` read parquet folders [datafusion]
comphead opened a new issue, #16460:
URL: https://github.com/apache/datafusion/issues/16460
> Create test data
> ```
> ls -la /tmp/t1
>
> -rw-r--r--@ 1 xxx wheel 12 Jun 6 08:35 .part-00000-e248d995-5eac-404e-a2ed-0eb16e27c005-c000.snappy.parquet.crc
> -rw-r--r--@ 1 xxx wheel 455 Jun 6 08:35 part-00000-e248d995-5eac-404e-a2ed-0eb16e27c005-c000.snappy.parquet
> ```
>
> - should realize this is a folder and rewrite as `/tmp/t1/*.parquet`
> ```
> DataFusion CLI v47.0.0
> CREATE EXTERNAL TABLE t1 STORED AS PARQUET location '/tmp/t1';
> Parquet error: Parquet error: Invalid Parquet file. Corrupt footer
> ```
>
_Originally posted by @comphead in [#13456](https://github.com/apache/datafusion/issues/13456#issuecomment-2949696136)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion` read parquet folders if non parquet files exists [datafusion]
Posted by "hendrikmakait (via GitHub)" <gi...@apache.org>.
hendrikmakait commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2992553343
As a side note, the docs at https://github.com/apache/datafusion/blob/a91e0421ebadf3a155508e28e272f5fb8356bca1/docs/source/user-guide/cli/datasources.md#parquet don't match the current behavior (no trailing `/` is needed) and will need further updates after implementing this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion` read parquet folders if non parquet files exists [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #16460: Make `datafusion` read parquet folders if non parquet files exists
URL: https://github.com/apache/datafusion/issues/16460
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion` read parquet folders if non parquet files exists [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2992156359
@hendrikmakait you can create a non-parquet file in the test, using `write_csv` or just create a random `txt` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion-cli` read parquet folders [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2988809219
Surprisingly if I try same tests in DF itself using unit or `slt` tests, it works. Looks like the issue in datafusion-cli only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion` read parquet folders if non parquet files exists [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2992154723
I played little bit yesterday in https://github.com/apache/datafusion/pull/16487
Feel free to explore, hope it helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion-cli` read parquet folders [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2992145520
good catch, putting another file to the folder crashes tests as well.
I'll update the header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion-cli` read parquet folders [datafusion]
Posted by "hendrikmakait (via GitHub)" <gi...@apache.org>.
hendrikmakait commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2991018757
That's what it looks like, I couldn't reproduce the problem with a directory that only contains `.parquet` files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion-cli` read parquet folders [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2991000231
I suspect datafusion-cli tried to read `.crc` as a parquet file...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion-cli` read parquet folders [datafusion]
Posted by "hendrikmakait (via GitHub)" <gi...@apache.org>.
hendrikmakait commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2988812320
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make `datafusion` read parquet folders if non parquet files exists [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #16460:
URL: https://github.com/apache/datafusion/issues/16460#issuecomment-2998350932
@hendrikmakait sorry I took the liberty to wrap my PR up, please feel free to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org