You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/08/26 21:03:21 UTC
[GitHub] [iceberg] samarthjain opened a new pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
samarthjain opened a new pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388
…
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388#discussion_r479427662
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -690,4 +692,25 @@ private ParquetReadBuilder(org.apache.parquet.io.InputFile file) {
return new ParquetReadSupport<>(schema, readSupport, callInit, nameMapping);
}
}
+
+ /**
+ * @param inputFiles an {@link Iterable} of parquet files. The order of iteration determines the order in which
+ * content of files are read and written to the @param outputFile
+ * @param outputFile the output parquet file containing all the data from @param inputFiles
+ * @param rowGroupSize the row group size to use when writing the @param outputFile
+ * @param schema the schema of the data
+ * @param metadata extraMetadata to write at the footer of the @param outputFile
+ */
+ public static void concat(Iterable<File> inputFiles, File outputFile, int rowGroupSize, Schema schema,
+ Map<String, String> metadata) throws IOException {
+ OutputFile file = Files.localOutput(outputFile);
+ ParquetFileWriter writer = new ParquetFileWriter(
+ ParquetIO.file(file), ParquetSchemaUtil.convert(schema, "table"),
+ ParquetFileWriter.Mode.CREATE, rowGroupSize, 0);
Review comment:
We can use the default row group size from table properties here. It will be ignored when appending files because row groups are appended directly and not rewritten.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388#discussion_r478768220
##########
File path: spark/src/test/java/org/apache/iceberg/spark/data/parquet/vectorized/TestParquetDictionaryEncodedVectorizedReads.java
##########
@@ -39,4 +51,48 @@
public void testVectorizedReadsWithNewContainers() throws IOException {
}
+
+ @Test
+ public void testMixedDictionaryNonDictionaryReads() throws IOException {
+ Schema schema = new Schema(SUPPORTED_PRIMITIVES.fields());
+
+ File dictionaryEncodedFile = temp.newFile();
+ Assert.assertTrue("Delete should succeed", dictionaryEncodedFile.delete());
+ Iterable<GenericData.Record> dictionaryEncodableData = RandomData.generateDictionaryEncodableData(
+ schema,
+ 10000,
+ 0L,
+ RandomData.DEFAULT_NULL_PERCENTAGE);
+ try (FileAppender<GenericData.Record> writer = getParquetWriter(schema, dictionaryEncodedFile)) {
+ writer.addAll(dictionaryEncodableData);
+ }
+
+ File plainEncodingFile = temp.newFile();
+ Assert.assertTrue("Delete should succeed", plainEncodingFile.delete());
+ Iterable<GenericData.Record> nonDictionaryData = RandomData.generate(schema, 10000, 0L,
+ RandomData.DEFAULT_NULL_PERCENTAGE);
+ try (FileAppender<GenericData.Record> writer = getParquetWriter(schema, plainEncodingFile)) {
+ writer.addAll(nonDictionaryData);
+ }
+
+ File mixedFile = temp.newFile();
+ Assert.assertTrue("Delete should succeed", mixedFile.delete());
+ OutputFile outputFile = Files.localOutput(mixedFile);
+ int rowGroupSize = Integer.parseInt(PARQUET_ROW_GROUP_SIZE_BYTES_DEFAULT);
+ ParquetFileWriter writer = new ParquetFileWriter(
Review comment:
What about adding a `Parquet.concat` util method? I don't think it is a good idea to make `ParquetIO` public just for this test case. But it would be nice to have a `concat` method somewhere that could concatenate Parquet files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388#issuecomment-682261042
+1 overall. I'd prefer not to expose `ParquetIO`, but if you think that building a `concat` helper is too much work for this PR, then we can do it in a follow-up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388#issuecomment-682908109
Thanks for the quick fix, @samarthjain! Nice work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1388: [Parquet Vectorized Reads] Fix reading of files with mix of dictionary and non-dictionary encoded row groups
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1388:
URL: https://github.com/apache/iceberg/pull/1388#discussion_r479428214
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java
##########
@@ -690,4 +692,25 @@ private ParquetReadBuilder(org.apache.parquet.io.InputFile file) {
return new ParquetReadSupport<>(schema, readSupport, callInit, nameMapping);
}
}
+
+ /**
+ * @param inputFiles an {@link Iterable} of parquet files. The order of iteration determines the order in which
+ * content of files are read and written to the @param outputFile
+ * @param outputFile the output parquet file containing all the data from @param inputFiles
+ * @param rowGroupSize the row group size to use when writing the @param outputFile
+ * @param schema the schema of the data
+ * @param metadata extraMetadata to write at the footer of the @param outputFile
+ */
+ public static void concat(Iterable<File> inputFiles, File outputFile, int rowGroupSize, Schema schema,
Review comment:
I think the input files and output file should use `InputFile` and `OutputFile`. That way this isn't limited to just the local FS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org