You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "abhishekd0907 (via GitHub)" <gi...@apache.org> on 2024/05/02 06:54:54 UTC

[PR] PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method [parquet-mr]

abhishekd0907 opened a new pull request, #1340:
URL: https://github.com/apache/parquet-mr/pull/1340

   ### Issue
   
   toParquetMetadata method converts org.apache.parquet.hadoop.metadata.ParquetMetadata to org.apache.parquet.format.FileMetaData but this does not set the dictionary page offset bit in FileMetaData.
   
   When a FileMetaData object is serialized while writing to the footer and then deserialized, the dictionary offset is lost as the dictionary page offset bit was never set.
   
   [PARQUET-1850](https://issues.apache.org/jira/browse/PARQUET-1850)  tried to fix this but it did only a partial fix.
   
   It sets setDictionary_page_offset only if getEncodingStats are present
   
   ```
   if (columnMetaData.getEncodingStats() != null
   && columnMetaData.getEncodingStats().hasDictionaryPages())
   { metaData.setDictionary_page_offset(columnMetaData.getDictionaryPageOffset()); } 
   ```
   ### Fix
   
   However, it should setDictionary_page_offset even when getEncodingStats are not present but encodings are present.
   
   It should use the implementation in ColumnChunkMetatdata below:
   ```
   public boolean hasDictionaryPage() {
   EncodingStats stats = getEncodingStats();
   if (stats != null) { 
   return stats.hasDictionaryPages() && stats.hasDictionaryEncodedPages(); 
   }
   
   Set<Encoding> encodings = getEncodings();
   return (encodings.contains(PLAIN_DICTIONARY) || encodings.contains(RLE_DICTIONARY));
   } 
   ```
   So new change in ParquetMetadataCOnvertor should be like:
   
   ```
   if (columnMetaData.hasDictionaryPage()) { metaData.setDictionary_page_offset(columnMetaData.getDictionaryPageOffset()); }
   ```
   
   ### Test
   
   Added a new UT for this scenario. All existing UTs should also pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1340:
URL: https://github.com/apache/parquet-java/pull/1340#issuecomment-4805346549

   Closing this pull request due to at least 3 months of inactivity. If you would like to continue the work, please feel free to reopen this pull request or open a new one. Thank you for your contribution to Apache Parquet!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #1340: PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method 
URL: https://github.com/apache/parquet-java/pull/1340


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method [parquet-mr]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #1340:
URL: https://github.com/apache/parquet-mr/pull/1340#issuecomment-2095077219

   ```
   Error:  Failed to execute goal com.diffplug.spotless:spotless-maven-plugin:2.30.0:check (default) on project parquet-hadoop: The following files had format violations:
   Error:      src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java
   Error:          @@ -1277,9 +1277,8 @@
   Error:           ····return·createParquetMetaData(dicEncoding,·dataEncoding,·true);
   Error:           ··}
   Error:           
   Error:          -
   Error:          -··private·static·ParquetMetadata·createParquetMetaData(Encoding·dicEncoding,·Encoding·dataEncoding,
   Error:          -·······················································boolean·includeDicStats)·{
   Error:          +··private·static·ParquetMetadata·createParquetMetaData(
   Error:          +······Encoding·dicEncoding,·Encoding·dataEncoding,·boolean·includeDicStats)·{
   Error:           ····MessageType·schema·=·parseMessageType("message·schema·{·optional·int32·col·(INT_32);·}");
   Error:           ····org.apache.parquet.hadoop.metadata.FileMetaData·fileMetaData·=
   Error:           ········new·org.apache.parquet.hadoop.metadata.FileMetaData(schema,·new·HashMap<String,·String>(),·null);
   Error:  Run 'mvn spotless:apply' to fix these violations.
   Error:  -> [Help 1]
   ```
   
   Please make the CI happy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2464: Fix dictionaryPageOffset flag setting in toParquetMetadata method [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1340:
URL: https://github.com/apache/parquet-java/pull/1340#issuecomment-4550162610

   This pull request has been automatically marked as stale because it has had no activity for at least 2 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Parquet!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org