You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/04/21 19:55:53 UTC
[GitHub] [incubator-iceberg] rdblue opened a new pull request #947: Spec: Add equality delete files
rdblue opened a new pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r412632887
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
So we need to infer the schema for equality delete files? Why not use a list or map that contains the assignment and has same schema?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r414736659
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
Yes, that sounds correct to me. I think a table may have more than one delete pattern, but we can detect this from the file schema at read time. When we open the equality delete file, we'll get the schema it uses and build a filter based on that.
Alternatively, we could add an ID set to the manifest file where this is tracked, but I try to keep columns in the manifest to a minimum. The main advantage of this approach that I can think of is that we'd be able to know the required projection for a data file before opening the delete files. I think this minor, so we'll see if we need it. We can always add a column ID list later if we need to since it is a compatible change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r413999612
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
The schema of a delete file is a subset of the table schema: "Equality delete files store any subset of a table's columns and use the table's field IDs." I'll update the wording to make that more clear.
I don't think it's a good idea to use the approach suggested on #360. Using map or list to hold values from multiple columns will mix data types, requiring conversion to binary and losing columnar benefits. Also, the use cases we've discussed use a stable set of columns; upsert by ID uses the ID columns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r415251617
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
Sounds great to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r413438836
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
There is no definition about schema here and as described below it looks like we will have different schemas for delete files. So I 'm suggesting use the schema discussed in [issue](https://github.com/apache/incubator-iceberg/issues/360#issuecomment-604867832).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r414279195
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
Thanks for the explanation for not choosing the list/map, that makes sense to me.
I know the fields should come from the data table schema, the question is how to get schema when reading these delete files as different tables would have different delete file schema as described. Sorry for not making the question clear.
So per your last comment, I guess the idea is that each table has a specific delete pattern which means a specific delete file schema. For example, a user table would have the delete pattern `id = 1000` which delete file schema is `1: id bigint`, a store_sales table would have the delete pattern `ss_store_sk = 10 and ss_customer_sk = 100` which delete file schema is `1: ss_store_sk bigint, 2: ss_customer_sk bigint`. Please help to correct me if I go in the wrong direction.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #947: Spec: Add equality delete files
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #947:
URL: https://github.com/apache/incubator-iceberg/pull/947#discussion_r413160433
##########
File path: site/docs/spec.md
##########
@@ -433,8 +433,17 @@ The rows in the delete file must be sorted by `file_path` then `position` to opt
* Sorting by `file_path` allows filter pushdown by file in columnar storage formats.
* Sorting by `position` allows filtering rows while scanning, to avoid keeping deletes in memory.
-Though the delete files can be written using any supported data file format in Iceberg, it is recommended to write delete files with same file format as the table's file format.
+Position-based delete files can be written using any supported data file format in Iceberg, but it is recommended to write delete files with same file format as the table's default file format.
+#### Equality Delete Files
+
+Equality delete files identify rows in a collection of data files that have been deleted by encoding equality predicates. Rows may be identified by more than one column.
Review comment:
I'm not sure I understand what you're suggesting.
This doesn't require schema inference. The delete file encodes the schema it was written with, along with the column IDs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org