You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rustyconover (via GitHub)" <gi...@apache.org> on 2023/08/01 21:17:22 UTC
[GitHub] [iceberg] rustyconover opened a new pull request, #8204: Python: add support for returning empty files in scans if requested.
rustyconover opened a new pull request, #8204:
URL: https://github.com/apache/iceberg/pull/8204
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8204:
URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1661654327
@rustyconover Do you have more context on it? What's the use case of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rustyconover commented on pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "rustyconover (via GitHub)" <gi...@apache.org>.
rustyconover commented on PR #8204:
URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1666912399
Hello @Fokko,
Sometimes, applications need to verify the presence of files based on specific filter expressions. For instance, if you're interested in checking for file membership in Iceberg, particularly for files with a `date > 2023-01-01`, utilizing the scan planner can be quite beneficial. This approach allows you to avoid loading unnecessary manifest files, enhancing efficiency.
Although working with manifest data provides valuable insights, it lacks the capability to filter using expressions. This is where the scan planner shines, enabling you to efficiently narrow down your results.
In my experience, I frequently encounter Iceberg tables containing an extensive number of data files, often exceeding 80,000. Therefore, any functionality that aids in filtering is immensely valuable.
Rusty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8204:
URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1666848345
Interesting, thanks for the explanation. How are you writing these files? Iceberg its focus is minimizing IO. Returning a file that doesn't contain any data, should be skipped for that reason. I don't recall anyone looking for such a feature on the Java side. @rdblue any thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #8204:
URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1666853305
@rustyconover Wouldn't it make more sense to go through the manifest instead of the actual scan planning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #8204:
URL: https://github.com/apache/iceberg/pull/8204#issuecomment-1666955823
Seems fine to me. Thanks, @rustyconover!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #8204: Python: add support for returning empty files in scans if requested.
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #8204:
URL: https://github.com/apache/iceberg/pull/8204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org