You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/06/06 13:54:23 UTC
[I] Make it easier to query parquet files on remote storage with `datafusion-cli` [datafusion]
alamb opened a new issue, #16304:
URL: https://github.com/apache/datafusion/issues/16304
### Is your feature request related to a problem or challenge?
- Part of https://github.com/apache/datafusion/issues/13456
There are more and more blogs like [this](https://altinity.com/blog/the-future-has-arrived-parquet-on-iceberg-finally-outperforms-mergetree) that show examples of running queries against data on remote object store
I would like to compare the performance of DataFusion to these other systems, but I find it really hard to run the examples
For example, in the above blog post,
```sql
INSERT INTO tripdata
SELECT * FROM s3('s3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet/*.parquet', NOSIGN)
SETTINGS max_threads=32, max_insert_threads=32, input_format_parallel_parsing=0;
```
When I try to follow the example in https://datafusion.apache.org/user-guide/cli/datasources.html#remote-files-directories to look at this same data in `datafusion-cli` it doesn't work (and it gives me a confusing message)
```shell
$ datafusion-cli
DataFusion CLI v47.0.0
> select count(*) from 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet';
Error during planning: table 'datafusion.public.s3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet' not found
>
```
I also tried setting `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` as suggested and it still fails:
```shell
$AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar datafusion-cli
DataFusion CLI v47.0.0
> select count(*) from 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet';
Error during planning: table 'datafusion.public.s3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet' not found
```
`CREATE EXTERNAL TABLE` does appear to work
```sql
> CREATE EXTERNAL TABLE hits
STORED AS PARQUET LOCATION 's3://altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet';
Object Store error: The operation lacked the necessary privileges to complete for path nyc_taxi_rides/data/tripdata_parquet: Error performing HEAD https://s3.us-east-1.amazonaws.com/altinity-clickhouse-data/nyc_taxi_rides/data/tripdata_parquet in 136.439542ms - Server returned non-2xx status code: 403 Forbidden:
```
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to query parquet files on remote storage with `datafusion-cli` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #16304: Make it easier to query parquet files on remote storage with `datafusion-cli`
URL: https://github.com/apache/datafusion/issues/16304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make it easier to query parquet files on remote storage with `datafusion-cli` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16304:
URL: https://github.com/apache/datafusion/issues/16304#issuecomment-2949347594
Actually, let's use https://github.com/apache/datafusion/issues/13456 to track this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org