[GitHub] [parquet-mr] ggershinsky commented on a change in pull request #925: PARQUET-2078: Failed to read parquet file after writing with the same …

[GitBox <gi...@apache.org>]

ggershinsky commented on a change in pull request #925:
URL: https://github.com/apache/parquet-mr/pull/925#discussion_r699959533



##########
File path: parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -201,9 +201,23 @@ public FileMetaData toParquetMetadata(int currentVersion, ParquetMetadata parque
     List<BlockMetaData> blocks = parquetMetadata.getBlocks();
     List<RowGroup> rowGroups = new ArrayList<RowGroup>();
     long numRows = 0;
+    long preBlockStartPos = 0;
+    long preBlockCompressedSize = 0;
     for (BlockMetaData block : blocks) {
       numRows += block.getRowCount();
+      long blockStartPos = block.getStartingPos();
+      // first block
+      if (blockStartPos == 4) {

Review comment:
       why this check is necessary, doesn't the first block always start at 4? Or this addresses a file merging usecase?

##########
File path: parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -1254,14 +1276,41 @@ static FileMetaData filterFileMetaDataByMidpoint(FileMetaData metaData, RangeMet
     return metaData;
   }
 
+  private static long tryUseFileOffset(RowGroup rowGroup, long preStartIndex, long preCompressedSize) {
+    long startIndex = 0;
+    startIndex = rowGroup.getFile_offset();
+    // skip checking the first rowGroup
+    // (in case of summary file, there are multiple first groups from different footers)
+    if (preStartIndex != 0 && preStartIndex <= startIndex) {
+
+      //calculate start index for other blocks
+      long minStartIndex = preStartIndex + preCompressedSize;
+      if (startIndex < minStartIndex) {
+        // a bad offset detected, try first column's offset
+        // can not use minStartIndex in case of padding
+        startIndex = getOffset(rowGroup.getColumns().get(0));

Review comment:
       this will throw an exception in encrypted files. 

##########
File path: parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
##########
@@ -201,9 +201,23 @@ public FileMetaData toParquetMetadata(int currentVersion, ParquetMetadata parque
     List<BlockMetaData> blocks = parquetMetadata.getBlocks();
     List<RowGroup> rowGroups = new ArrayList<RowGroup>();
     long numRows = 0;
+    long preBlockStartPos = 0;
+    long preBlockCompressedSize = 0;
     for (BlockMetaData block : blocks) {
       numRows += block.getRowCount();
+      long blockStartPos = block.getStartingPos();
+      // first block
+      if (blockStartPos == 4) {
+        preBlockStartPos = 0;
+        preBlockCompressedSize = 0;
+      }
+      if (preBlockStartPos != 0) {
+        assert blockStartPos >= preBlockStartPos + preBlockCompressedSize;

Review comment:
       why >= instead of == ? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org