You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/22 06:16:47 UTC
[GitHub] [iceberg] LionTao opened a new issue, #6481: Support for predicate pushdown on s3
LionTao opened a new issue, #6481:
URL: https://github.com/apache/iceberg/issues/6481
### Query engine
None, I'm using Java API
### Question
Currently, when reading files, iceberg utilizes vectorized reading to read files.
But when a filter is applied, we can use the s3 select API to reduce the file size needed to transfer. That will reduce time and cost.
According to s3 documents and minio documents, parquet is the only supported format for iceberg.
Any guides or advice on implementing this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] LionTao commented on issue #6481: Support for predicate pushdown on s3
Posted by GitBox <gi...@apache.org>.
LionTao commented on issue #6481:
URL: https://github.com/apache/iceberg/issues/6481#issuecomment-1364677516
Hi, thank you for your reply, and merry christmas🎄
> Could you elaborate a bit more on what you would like to do here?
**I am building a distributed computing engine for trajectory data based on Faas service, Dapr and Iceberg. I want to minimize the network IO. So I think using S3-Select may be helpful.**
After digging around the code, I found currently the FileIO API relies on Stream to process content. But the S3-select response seems to be different.
So I think if we need to utilize the S3-Select ability, we need some new mechanism to deal with Input files.
Also, this new mechanism may not be supported by all storage, so it could be AWS-specific. I'm not sure whether it is ideal for the extensibility of Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6481: Support for predicate pushdown on s3
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6481:
URL: https://github.com/apache/iceberg/issues/6481#issuecomment-1626379287
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #6481: Support for predicate pushdown on s3
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6481: Support for predicate pushdown on s3
URL: https://github.com/apache/iceberg/issues/6481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #6481: Support for predicate pushdown on s3
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #6481:
URL: https://github.com/apache/iceberg/issues/6481#issuecomment-1363228278
Could you elaborate a bit more on what you would like to do here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #6481: Support for predicate pushdown on s3
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6481:
URL: https://github.com/apache/iceberg/issues/6481#issuecomment-1605176670
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org