You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/10/09 04:06:11 UTC

[GitHub] [iceberg] shangxinli opened a new pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

shangxinli opened a new pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566


   Issue link: https://github.com/apache/iceberg/issues/193


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505895293



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);

Review comment:
       Yes, we set it true by default. But eventually, it will rely on the filter we pushed here skip pages. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505861612



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       After reading more about Parquet and page skipping, I don't think my comment above holds true in the general case. Seems like `nextRowGroup` counter should still accurately depict the current row group being processed. The only edge case that I see is in https://github.com/apache/parquet-mr/blob/fe9ab5d22c602776f44b027ad3cc05609eddba54/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L967 where a whole row group may be skipped if the column index allows all the pages in the row group to be skipped. In such a case, `readNextFilteredRowGroup` will automatically advance to the next block without our knowledge and hence without the counter being updated. This will be an issue for all further row group skipping and row postition calculations in Iceberg's ParquetReader. However, I am not sure if there can be such a case where Iceberg's row group skipping is not able to skip the row group, but Parquet's page skipping ends put skipping all page and consequ
 ently the whole row group. 

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();

Review comment:
       I read more into the code and I think I understand the benefit here. So a Parquet row group is made up of multiple pages, previously we were only skipping unmatched row groups, now we can skip unmatched row groups and unmatched pages. Let me know if this seems correct.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505847585



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);

Review comment:
       Seems like what we want here is `useColumnIndexFilter` and not `useRecordFilter` else `readNextFilteredRowGroup()` is equivalent to `readNextRowGroup()` https://github.com/apache/parquet-mr/blob/fe9ab5d22c602776f44b027ad3cc05609eddba54/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L956




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-712292673


   I am new to Iceberg. Can you add more details about 'jmh'? My plan is to have unit tests along with this PR. I can then open another PR if benchmarking tests and other tests are needed.  
   
   > Nice work @shangxinli! Do you plan to add any jmh benchmark tests to illustrate the filtering benefits?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r513490614



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       @shardulm94, I talked to the Parquet community(PARQUET-1927) and it seems we don't need Parquet change anymore. What we can do is to use ParquetFileReader constructor's filter(page level stats, row group stats, dictionray and future bloomfilter). All the following calls like reader.getRowGroups() and reader.getFilteredRecordCount() in ReadConf are all filtered values. It also simplifies the ReadConf constructor and ParquetReader/VectorizedParquetReader. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-709599748


   I guess https://github.com/apache/iceberg/blob/cd70cac279d3f14ba61f0143f9988d4cc9413651/parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java#L120 also needs to be changed to use `valuesRead + skippedValues` just like `hasNext()`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505911443



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);
+          optionsBuilder.withRecordFilter(ParquetFilters.convert(schema, filter, caseSensitive));

Review comment:
       We don't have to move it to the absolute top, we can only fetch the file schema if the filter is supposed to be used. It will lead to an extra open call, but I am not sure I see a good way around it. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721809179


   @rdblue, make sense. Now the question is for ColumnIndex filter do we also want to reimplement? And the same question for future bloomfilter. Since we already had other types of filter reimplemented,  it makes more sense to do the same thing. It is just more effort to do, but I can do it if Iceberg decides to reimplement all the filters in Parquet. 
   
   If we don't reimplement, we might need Parquet to release 1.11.2(we discussed in the last Parquet meeting and that is possible if needed) with a fix to have skipped record count. But we still have the same issue you mentioned above(IN, STARTWITH not supported etc). 
   
   @shardulm94, I didn't address your comments yet. If we decide to reimplement the ColumnIndex filter, then we cannot call readNextRowGroupFilter(). Instead, we will filter pages(maybe in BaseColumnIterator?, need to look into it more). So I will address later once we have agreement here.  
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721319133


   One more thing, we also wanted to use Iceberg's comparators. I think we had those done before Parquet introduced its comparators, and Iceberg also has strong support for `CharSequence` without conversion to `String`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505914331



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       This edge case is probably possible with a carefully crafted file. e.g. consider the min/max values of single column
   Row Group: `[1, 1000]`
   Pages: `[[1, 10], [900, 1000]]`
   And expression `column = 500` will match the row group but will not match any of the pages resulting in the next row group being read automatically without the `nextRowGroup` counter being updated




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r516902119



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -623,17 +630,10 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
       if (filter != null) {
         // TODO: should not need to get the schema to push down before opening the file.
         // Parquet should allow setting a filter inside its read support
-        MessageType type;

Review comment:
       The comment above sounds like another possible reason why we wanted to reimplement filters. Tailoring the filter to each file is difficult, compared to evaluating the same filter for a file.
   
   If I remember correctly, the need to tailor the filter for the file is because we use id-based column resolution. So the file might contain `1: a int` and Iceberg has a filter for `1: x long`. Unless the filter is translated to use `a` instead of `x`, Parquet will skip the file because it doesn't think the column exists (and is all nulls).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-707255509


   FYI @aokolnychyi, @prodeezy: this PR adds support for page-level skipping in Parquet.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721874701


   @shangxinli, I think we probably want to build the filtering into Iceberg. I could be convinced otherwise, but it would take a lot of work to use Parquet filters.
   
   After my comment yesterday, I was looking at the implementation here and [noticed another problem](https://github.com/apache/iceberg/pull/1566#discussion_r516902119): Parquet expects a filter that references columns using the name that was written into each data file. Iceberg allows renaming and so the Iceberg filters are written to use field IDs instead of names. That is needed for correctness. If I rename field `a` to `x`, then I have to rewrite the filter `x > 5` to `a > 5` when reading an old data file, or else Parquet thinks there are no matching records because column `x` is not present and is all nulls.
   
   Parquet also doesn't support customizing the filter in `ReadSupport`, so to do this you have to read each file's footer to get its schema, rewrite the filter, and then open the file again to read it.
   
   I'm remembering more of the issues with the original filter path now, and I think that it makes more sense to do the work in Iceberg for now, with the intent to get the work upstream in a 2.0 Parquet API that fixes some of these issues.
   
   What do you think?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505964469



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);
+          optionsBuilder.withRecordFilter(ParquetFilters.convert(schema, filter, caseSensitive));

Review comment:
       Yes, we call it when we need to convert. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721899321


   @rdblue I think it makes sense.  


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r506766877



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       This is such a critical comment! That is absolutely possible and we need to handle that case. I looked at the code again and found even the existing code, we are missing a condition check 'pages == null'. It seems valid that readNextRowGroup() can return null https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L899. In that case, the caller needs to handle while it doesn't. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli edited a comment on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli edited a comment on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721292039


   @shardulm94 I am also curious about why Iceberg reimplements those filters. @rdblue Can you cast some light on this? 
   
   I can see the pros and cons of reimplementing. The concern for reimplementing is that it creates duplication and fragments. For short-term workaround it is fine, but for the long run, I think deduping to one implementation makes more sense. Otherwise, when new filters are added in Parquet, we need to reimplement them here. Column index is one, and bloom filer is another one on the way, and so on. If some filters in Iceberg but not in Parquet, we can bring in to the Parquet community to add them. What do you think?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r506766877



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       This is such a critical comment! That is absolutely possible and we need to handle that case. I looked at the code again and found even the existing code, we are missing a condition check 'pages == null'. It seems valid that readNextRowGroup() can return null https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L899. In that case, the caller needs to handle while it doesn't. It is more complicated for readNextFilteredRowGroup() because it can advance internally without we know. I will add some handling code there. If you have suggestions, let me know. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721292039


   @shardulm94 I am also curious about why Iceberg reimplements those filters. @rdblue Can you cast some light on this? 
   
   I can see the pros and cons of reimplementing. The concern for reimplementing is that it creates duplication and fragments. For short-term workaround it is fine, but for the long run, I think deduping to one implementation makes more sense. When new filters are added in Parquet, we need to reimplement them here. Column index is one, and bloom filer is another one on the way, and so on. If some filters in Iceberg but not in Parquet, we can bring in to the Parquet community to add them. What do you think?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505912203



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(
+          Type.TypeID.BOOLEAN,
+          Type.TypeID.INTEGER,
+          Type.TypeID.LONG,
+          Type.TypeID.FLOAT,
+          Type.TypeID.DOUBLE,
+          Type.TypeID.DATE,
+          Type.TypeID.TIME
+  );
+
   private ParquetFilters() {
   }
 
+  public static boolean isSupportedFilter(Expression expr, Schema schema, boolean caseSensitive) {

Review comment:
       Gotcha! Since this is not in the critical loop of reading rows, I think the cost of catching exception should not be significant in the grand scheme of things. I don't have a strong opinion for this though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505691518



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(

Review comment:
       Any reason why other types like `TIMESTAMP`, `STRING` and other binary columns are not supported? Seems like `ParquetFilters` is able to convert expressions for these types too.

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&

Review comment:
       What is the rationale behind `schema.getAliases() != null` check?

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);
+          optionsBuilder.withRecordFilter(ParquetFilters.convert(schema, filter, caseSensitive));

Review comment:
       I think the first argument to `ParquetFilters.convert` should be the file schema and not the table schema. Since Iceberg allows column renaming, a column in the table schema maybe named differently in the file. If you look at https://github.com/apache/iceberg/blob/a238a90eb987bfbf5d14b5cab8d109e53a75e861/parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java#L120 it will use field ids to get the corresponding column name from the provided schema (file). You can also see this pattern at https://github.com/apache/iceberg/blob/a238a90eb987bfbf5d14b5cab8d109e53a75e861/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L628

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(
+          Type.TypeID.BOOLEAN,
+          Type.TypeID.INTEGER,
+          Type.TypeID.LONG,
+          Type.TypeID.FLOAT,
+          Type.TypeID.DOUBLE,
+          Type.TypeID.DATE,
+          Type.TypeID.TIME
+  );
+
   private ParquetFilters() {
   }
 
+  public static boolean isSupportedFilter(Expression expr, Schema schema, boolean caseSensitive) {

Review comment:
       Rather than introducing more code to check if the filter is supported, can we just eagerly try to convert filter and use the exception as a signal that the filter is unsupported?

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();

Review comment:
       Iceberg is evaluating filters for row groups itself and skipping them (just a few lines before this), how does this change improve upon what Iceberg already does? (I am not too familiar with Parquet, so I may be totally missing the point)

##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       I am not sure if the `nextRowGroup` counter and consequently `rowPosition` will be accurate in case of skipped row groups. Can you add a test to `TestSparkParquetReadMetadataColumns` which would invoke page skipping?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505886885



##########
File path: parquet/src/test/java/org/apache/iceberg/parquet/TestParquetColumnIndex.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.parquet;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.avro.generic.GenericData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.parquet.Parquet.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.Files.localInput;
+import static org.apache.iceberg.TableProperties.PARQUET_PAGE_SIZE_BYTES;
+import static org.apache.iceberg.parquet.ParquetWritingTestUtils.writeRecords;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+
+public class TestParquetColumnIndex {
+
+  private MessageType parquetSchema = new MessageType("schema",
+      new PrimitiveType(REQUIRED, INT32, "intCol"));
+  private Schema schema = ParquetSchemaUtil.convert(parquetSchema);
+
+  @Rule
+  public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testColumnIndexFilter() throws IOException {
+    File parquetFile = generateFileWithMultiplePages(ParquetAvroWriter::buildWriter);
+    int totalCount = getPageRecordCount(parquetFile, null);
+    int filterCount = getPageRecordCount(parquetFile,
+            Expressions.and(Expressions.notNull("intCol"), Expressions.equal("intCol", 1)));
+    Assert.assertTrue(filterCount < totalCount);

Review comment:
       Shouldn't this end up reading one and only one page? Can make the assertion more strict?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505901768



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);
+          optionsBuilder.withRecordFilter(ParquetFilters.convert(schema, filter, caseSensitive));

Review comment:
       Do you recommend to move "type = schemaReader.getFileMetaData().getSchema()" to the beginning of this method so that we can use the file schema instead of table schema? I worry it could result in more storage API calls because when the condition "if (readerFunc != null || batchedReaderFunc != null)" is true, we don't have this API call today.   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] prodeezy commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
prodeezy commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-712294533


   > I am new to Iceberg. Can you add more details about 'jmh'? My plan is to have unit tests along with this PR. I can then open another PR if benchmarking tests and other tests are needed.
   
   JMH is a toolkit for microbenchmarks in Java. There are some tests here in Iceberg that use it:
   https://github.com/apache/iceberg/tree/c9ccdfef4fd7798cd824047cf4e8077c31c1c9e1/spark2/src/jmh/java/org/apache/iceberg/spark/data/parquet
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] prodeezy commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
prodeezy commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-712289914


   Nice work @shangxinli! Do you plan to add any jmh benchmark tests to illustrate the filtering benefits?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505891344



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);

Review comment:
       I found that `useColumnIndexFilter` is true by default




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-721317727


   There are a few reasons why Iceberg reimplemented the filters.
   
   * Faster iteration and more features without needing Parquet releases
   * Parquet's filter API has some problems
     * Evaluation is negated (`canDrop` vs `shouldRead`), which has led to more bugs
     * It is missing some predicates that we need to be well supported, like `startsWith`, `in`, `alwaysTrue`, and `alwaysFalse`
     * The API is very difficult to work with
   * Iceberg replaces record materialization, so we would need to run these filters from Iceberg code anyway
   * Iceberg had already implemented similar filters, like stats evaluation, so it was simple to reuse that code
   
   To fix some of the issues with the Parquet API, my hope was that eventually Parquet would use Iceberg's expression API and filters in place of its own. We'd need to refactor a bit to make this happen, but I think it would still be a good option. There are several things that I think would be great to standardize across some of the storage projects like the `FileIO` classes and the expressions.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r513483116



##########
File path: parquet/src/test/java/org/apache/iceberg/parquet/TestParquetColumnIndex.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.parquet;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.avro.generic.GenericData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.parquet.Parquet.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.Files.localInput;
+import static org.apache.iceberg.TableProperties.PARQUET_PAGE_SIZE_BYTES;
+import static org.apache.iceberg.parquet.ParquetWritingTestUtils.writeRecords;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+
+public class TestParquetColumnIndex {
+
+  private MessageType parquetSchema = new MessageType("schema",
+      new PrimitiveType(REQUIRED, INT32, "intCol"));
+  private Schema schema = ParquetSchemaUtil.convert(parquetSchema);
+
+  @Rule
+  public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testColumnIndexFilter() throws IOException {
+    File parquetFile = generateFileWithMultiplePages(ParquetAvroWriter::buildWriter);
+    int totalCount = getPageRecordCount(parquetFile, null);
+    int filterCount = getPageRecordCount(parquetFile,
+            Expressions.and(Expressions.notNull("intCol"), Expressions.equal("intCol", 1)));
+    Assert.assertTrue(filterCount < totalCount);

Review comment:
       Yes. I changed to test code to compare with one single page count. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505891344



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);

Review comment:
       I found that `useColumnIndexFilter` is true by default, so we don't really need to specify it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505702358



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       I am not sure if the `nextRowGroup` counter and consequently `rowPosition` will be accurate in case of skipped row groups. Having the correct `rowPosition` relative to the start of the file is critical for applying row-level deletes. Can you add a test to `TestSparkParquetReadMetadataColumns` which would invoke page skipping?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505815237



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&

Review comment:
       If not, there will be a NPE thrown in https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java#L120. There was some discussion in https://github.com/apache/iceberg/issues/193 with Ryan. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505905559



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(
+          Type.TypeID.BOOLEAN,
+          Type.TypeID.INTEGER,
+          Type.TypeID.LONG,
+          Type.TypeID.FLOAT,
+          Type.TypeID.DOUBLE,
+          Type.TypeID.DATE,
+          Type.TypeID.TIME
+  );
+
   private ParquetFilters() {
   }
 
+  public static boolean isSupportedFilter(Expression expr, Schema schema, boolean caseSensitive) {

Review comment:
       That is what I originally use but I changed it because throwing/catching exception is pretty expensive. In the non-error case, we should avoid that, although I understand creating some separate code to keep tracking is another pain.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-723645208


   @rdblue I looked at the implementation of ColumnIndex. Here are what we need to do to rewrite this filter and use it in Iceberg. 
   1. Rewrite the equivalent filters in the Parquet ColumnIndexFilter class. 
   2. Rewrite ParquetFileReader#getRowRanges(), which applies the column index filters and get rowRanges (RowRanges).  
   3. Use the filtered rowRanges gotten in step 2 to build a PageReadStore like ColumnChunkPageReadStore https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L976. Unfortunately ColumnChunkPageReadStore() is not a public class for Iceberg to use. This would require us to rewrite our own PageReadStore, or let Parquet make it public. 
   4. With the filtered rowRanges to get filteredOffsetIndex, we can read all the parts needed with it. Then we need SynchronizingColumnReader to synchronize cross columns. Unfortunately, SynchronizingColumnReader is not a public class either. We need to rewrite it in Iceberg.
   5. Wrap up the step #2, #3, #4 into a method to replace readNextFilteredRowGroup() in Parquet. 
   
   The work seems more than just rewriting the filter but also rewriting ColumnChunkPageReadStore, SynchronizingColumnReader, and partially ParquetFileReader. This makes me think should we continue rewriting or reusing? If we choose to reuse, we can limit the filter not to support startWith, in.., or we can add them into Parquet. Let me know you thought.  
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505895293



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&
+                ParquetFilters.isSupportedFilter(filter, schema, caseSensitive)) {
+          optionsBuilder.useRecordFilter(filterRecords);

Review comment:
       Yes, we set it true by default. But eventually, it will rely on the filter we pushed here to skip pages. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505903613



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(

Review comment:
       Sorry I overlooked at UNSUPPORTED_TYPES defined in ExpressToSearchArgument.java. It is for ORC instead of Parquet. I will remove these. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#issuecomment-707255168


   Thanks for opening this PR, @shangxinli! I'll take a look at this soon. It's great to see you have it working.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505702358



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       I am not sure if the `nextRowGroup` counter and consequently `rowPosition` will be accurate in case of skipped pages. Having the correct `rowPosition` relative to the start of the file is critical for applying row-level deletes. Can you add a test to `TestSparkParquetReadMetadataColumns` which would invoke page skipping?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r506766877



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       This is such an important comment! That is absolutely possible and we need to handle that case. I looked at the code again and found even the existing code, we are missing a condition check 'pages == null'. It seems valid that readNextRowGroup() can return null https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java#L899. In that case, the caller needs to handle while it doesn't. It is more complicated for readNextFilteredRowGroup() because it can advance internally without we know. I will add some handling code there. If you have suggestions, let me know. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505886885



##########
File path: parquet/src/test/java/org/apache/iceberg/parquet/TestParquetColumnIndex.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.parquet;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.avro.generic.GenericData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.data.parquet.GenericParquetReaders;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.parquet.Parquet.ReadBuilder;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.Files.localInput;
+import static org.apache.iceberg.TableProperties.PARQUET_PAGE_SIZE_BYTES;
+import static org.apache.iceberg.parquet.ParquetWritingTestUtils.writeRecords;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+
+public class TestParquetColumnIndex {
+
+  private MessageType parquetSchema = new MessageType("schema",
+      new PrimitiveType(REQUIRED, INT32, "intCol"));
+  private Schema schema = ParquetSchemaUtil.convert(parquetSchema);
+
+  @Rule
+  public TemporaryFolder temp = new TemporaryFolder();
+
+  @Test
+  public void testColumnIndexFilter() throws IOException {
+    File parquetFile = generateFileWithMultiplePages(ParquetAvroWriter::buildWriter);
+    int totalCount = getPageRecordCount(parquetFile, null);
+    int filterCount = getPageRecordCount(parquetFile,
+            Expressions.and(Expressions.notNull("intCol"), Expressions.equal("intCol", 1)));
+    Assert.assertTrue(filterCount < totalCount);

Review comment:
       Shouldn't this end up reading one and only one page? Can make the assertion more strict?
   Based on my comment in https://github.com/apache/iceberg/pull/1566#discussion_r505847585 I doubt if the reduced row count is a result of page skipping or just row group skipping.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505884115



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&

Review comment:
       The Iceberg table schema will probably rarely have aliases. I think after my recommendation at https://github.com/apache/iceberg/pull/1566#discussion_r505689098 you should no longer see the NPE and this check can be removed.
   
   I think this is also in alignment with what @rdblue  says at https://github.com/apache/iceberg/issues/193#issuecomment-692390785
   > When converting to the Parquet FilterApi, the aliases must come from a Schema instance that was created from the MessageType of the file you're pushing filters to. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r507854338



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       @shardulm94, after looking into the code deeper, I think there could be two solutions here. 1) Change Parquet to let readNextFilteredGroup() return not only the filtered pages but also the record counts it skipped. 2) In the ReadConf constructor, when deciding 'shouldRead', we add ColumnIndex filter. This will ensure if a row group is called by readNextFilteredGroup(), there will always be at least 1 page returned and no implicitly advance internally without Iceberg knowing it. Here is the place for determining the 'shouldRead' boolean variable. https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java#L106
   
   For option 1, we will need to wait for the new release of Parquet. I am not sure releasing 1.10.2 is an option for Parquet. If not, we have to wait for Parquet 1.12.0. For option 2, it requires a pretty significant change. And we have to make sure the implementation here is consistent with Parquet' ColumnIndex filter implementation. 
   
   Please share your thought and we can discuss. 
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505966802



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -582,6 +582,13 @@ public ReadBuilder withNameMapping(NameMapping newNameMapping) {
           optionsBuilder = ParquetReadOptions.builder();
         }
 
+        if (filter != null &&
+                schema.getAliases() != null &&

Review comment:
       make sense




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r516903215



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ReadConf.java
##########
@@ -85,34 +84,28 @@
       this.projection = ParquetSchemaUtil.pruneColumnsFallback(fileSchema, expectedSchema);
     }
 
+    // ParquetFileReader has filters(stats, dictionary and future bloomfilter) in the constructor,
+    // so getRowGroups returns filtered row groups
     this.rowGroups = reader.getRowGroups();
-    this.shouldSkip = new boolean[rowGroups.size()];
 
     // Fetch all row groups starting positions to compute the row offsets of the filtered row groups
     Map<Long, Long> offsetToStartPos = generateOffsetToStartPos();
     this.startRowPositions = new long[rowGroups.size()];
 
-    ParquetMetricsRowGroupFilter statsFilter = null;
-    ParquetDictionaryRowGroupFilter dictFilter = null;
-    if (filter != null) {
-      statsFilter = new ParquetMetricsRowGroupFilter(expectedSchema, filter, caseSensitive);
-      dictFilter = new ParquetDictionaryRowGroupFilter(expectedSchema, filter, caseSensitive);
-    }

Review comment:
       This PR removes support for pushing down filters like `startsWith` and `in`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shangxinli commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505965911



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java
##########
@@ -38,9 +43,71 @@
 
 class ParquetFilters {
 
+  private static final Set<Operation> SUPPORTED_OPS = ImmutableSet.of(
+          Operation.IS_NULL,
+          Operation.NOT_NULL,
+          Operation.EQ,
+          Operation.NOT_EQ,
+          Operation.GT,
+          Operation.GT_EQ,
+          Operation.LT,
+          Operation.LT_EQ);
+
+  private static final Set<Type.TypeID> SUPPORTED_TYPES = ImmutableSet.of(
+          Type.TypeID.BOOLEAN,
+          Type.TypeID.INTEGER,
+          Type.TypeID.LONG,
+          Type.TypeID.FLOAT,
+          Type.TypeID.DOUBLE,
+          Type.TypeID.DATE,
+          Type.TypeID.TIME
+  );
+
   private ParquetFilters() {
   }
 
+  public static boolean isSupportedFilter(Expression expr, Schema schema, boolean caseSensitive) {

Review comment:
       We throw it once per file in case of unsupported filters. I don't have strong opinions either. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] shardulm94 commented on a change in pull request #1566: Parquet: Support Page Skipping in Iceberg Parquet Reader

Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1566:
URL: https://github.com/apache/iceberg/pull/1566#discussion_r505914331



##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetReader.java
##########
@@ -130,13 +139,23 @@ private void advance() {
 
       PageReadStore pages;
       try {
-        pages = reader.readNextRowGroup();
+        // Because of the issue of PARQUET-1901, we cannot blindly call readNextFilteredRowGroup()
+        if (hasRecordFilter) {
+          pages = reader.readNextFilteredRowGroup();
+        } else {
+          pages = reader.readNextRowGroup();
+        }
       } catch (IOException e) {
         throw new RuntimeIOException(e);
       }
 
+      long blockRowCount = blocks.get(nextRowGroup).getRowCount();
+      Preconditions.checkState(blockRowCount >= pages.getRowCount(),
+              "Number of values in the block, %s, does not great or equal number of values after filtering, %s",
+              blockRowCount, pages.getRowCount());
       long rowPosition = rowGroupsStartRowPos[nextRowGroup];

Review comment:
       This edge case is probably possible with a carefully crafted file. e.g. consider the min/max values of single column
   Row Group: `[1, 1000]`
   Pages: `[[1, 10], [900, 100]]`
   And expression `column = 500` will match the row group but will not match any of the pages resulting in the next row group being read automatically without the `nextRowGroup` counter being updated




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org