You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sweetpythoncode (via GitHub)" <gi...@apache.org> on 2023/03/06 15:54:27 UTC
[GitHub] [iceberg] sweetpythoncode opened a new issue, #7027: Iceberg add_files procedure with partition_filter scan non needed folders
sweetpythoncode opened a new issue, #7027:
URL: https://github.com/apache/iceberg/issues/7027
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
source structure example: `s3://bucket/data/id=123/name=test/date=321/result.orc`
```
CALL iceberg_catalog.system.add_files(
table => 'test.test_name',
source_table => '`orc`.`s3://bucket/data/`',
partition_filter => map('id', '3')
check_duplicate_files => false
```
`partition_filter` option does not handle the order of partition, which produces nested folders scanning until finding the first match. Should we run filter by partition in order before run nested `Listing leaf files and directories`?
**Example of current flow:**
```
s3://bucket/data/id=1/name=test/date=321/result.orc -> Listing leaf files and directories on each sub folder
s3://bucket/data/id=2/name=test/date=321/result.orc -> Listing leaf files and directories on each sub folder
s3://bucket/data/id=3/name=test/date=321/result.orc -> Match needed partition_filter ignore next folders
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg add_files procedure with partition_filter scan non needed folders [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-2345009886
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg add_files procedure with partition_filter scan non needed folders [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
URL: https://github.com/apache/iceberg/issues/7027
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1469032864
@sweetpythoncode yes, I have come across both issues identified. Partition listing can be improved and when a path is provided which is not the root path without partition info, then partition values are not extracted correctly. I am working on them and will let you know once I have a PR out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1498648732
@abmo-x Wow! Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg add_files procedure with partition_filter scan non needed folders [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-2313785851
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1471846184
@szehon-ho yes, I try, but it's not using the predicted filter in the dest path, It will anyway scan the full data before applying that filter, based on the code which I found in `getPartitions` method for `add_files` procedure(I also debug it to check if that true).
@abmo-x Sweet, wait for it, thanks you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg add_files procedure with partition_filter scan non needed folders [iceberg]
Posted by "siumingdev (via GitHub)" <gi...@apache.org>.
siumingdev commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1830151558
Will this be improved or may I have suggestions on alternative ways doing the equivalent? say using the Java API alone without Spark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] sweetpythoncode commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "sweetpythoncode (via GitHub)" <gi...@apache.org>.
sweetpythoncode commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1498581385
@abmo-x Hello, any progress on this one, thanks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1498621904
Hi @sweetpythoncode I was testing it right now :) hope to have a PR soon. Will keep you posted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1509611647
@sweetpythoncode
found the root cause and implemented the solution, its all ready to go. will publish the PR soon. FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #7027: Iceberg add_files procedure with partition_filter scan non needed folders
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #7027:
URL: https://github.com/apache/iceberg/issues/7027#issuecomment-1468891226
Did you try full partition filter? Also cc @abmo-x @dramaticlly , who are expert users of add_files, if they some experience. Its been awhile since I looked at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org