You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/15 11:39:03 UTC
[GitHub] [iceberg] szlta opened a new pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
szlta opened a new pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748
The Hive ORC vectorization code path considered the ORC file start/end positions as per given by the task.
In reality it should always read the OrcTail (metadata) parts from the end of the actual file,
irrespective of the task offset/length information.
This problem has come out after #3666 corrected that the split offsets provided by native file format appenders
should be considered in split generation. Unfortunately the unit tests were not populating split_offsets either earlier...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769577513
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
Review comment:
nit: would it work just as well with a slightly different value, like 190k or 210k? It was hard for me to eyeball the difference between this 200k value and the 20k value above when generating the records. Made me think for a moment that the values are related somehow :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769570953
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
+ // (Would request 5 if ORC split generation wouldn't be split (aka stripe) offset aware).
+ shell.getHiveConf().set(InputFormatConfig.SPLIT_SIZE, "200000");
+ List<Object[]> result = shell.executeStatement("SELECT * FROM targettab ORDER BY last_name");
+
+ Assert.assertEquals(20000, result.size());
+ } finally {
+ shell.getHiveConf().unset(InputFormatConfig.SPLIT_SIZE);
Review comment:
`HiveIcebergStorageHandlerTestUtils.init` supposed to take care of the config changes, and it is called `@Before`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769569528
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
Review comment:
Is there any easy way to check the number of the splits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769576474
##########
File path: hive3/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java
##########
@@ -98,11 +100,13 @@ private HiveVectorizedReader() {
try {
switch (format) {
case ORC:
- InputSplit split = new OrcSplit(path, null, task.start(), task.length(), (String[]) null, null,
- false, false, Lists.newArrayList(), 0, task.length(), path.getParent());
- RecordReader<NullWritable, VectorizedRowBatch> recordReader = null;
- recordReader = new VectorizedOrcInputFormat().getRecordReader(split, job, reporter);
+ OrcTail orcTail = VectorizedReadUtils.getOrcTail(inputFile, job);
Review comment:
Maybe it'd be useful to have a short comment on why the tail read is necessary here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szlta commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
szlta commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769593880
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
Review comment:
It's very deeply internal unfortunately..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769569090
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
+ // (Would request 5 if ORC split generation wouldn't be split (aka stripe) offset aware).
+ shell.getHiveConf().set(InputFormatConfig.SPLIT_SIZE, "200000");
Review comment:
Maybe use shell.setHiveSessionValue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marton-bod commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
marton-bod commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769576049
##########
File path: hive3/src/main/java/org/apache/iceberg/orc/VectorizedReadUtils.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.orc;
+
+import java.io.IOException;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.io.InputFile;
+import org.apache.orc.impl.OrcTail;
+import org.apache.orc.impl.ReaderImpl;
+
+/**
+ * Utilities that rely on Iceberg code from org.apache.iceberg.orc package.
Review comment:
nit: "that rely on org.apache.iceberg.orc.." AND has to do with vectorization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769568842
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
Review comment:
Maybe use `shell.setHiveSessionValue`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769571639
##########
File path: hive3/src/main/java/org/apache/iceberg/orc/VectorizedReadUtils.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.orc;
+
+import java.io.IOException;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.io.InputFile;
+import org.apache.orc.impl.OrcTail;
+import org.apache.orc.impl.ReaderImpl;
+
+/**
+ * Utilities that rely on Iceberg code from org.apache.iceberg.orc package.
+ */
+public class VectorizedReadUtils {
+
+ private VectorizedReadUtils() {
+
+ }
+
+ /**
+ * Opens the ORC inputFile and reads the metadata information to construct the OrcTail content.
+ * Unfortunately the API doesn't allow simple access to OrcTail, so we need the serialization trick.
+ * @param inputFile - the ORC file
+ * @param job - JobConf instance for the current task
+ * @throws IOException - errors relating to accessing the ORC file
+ */
+ public static OrcTail getOrcTail(InputFile inputFile, JobConf job)
Review comment:
nit: could this be a single line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary merged pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769601151
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
Review comment:
Sad :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#issuecomment-994794263
Let's give some time for others to review, if they think. Will merge tomorrow, if not other reviews arrive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#issuecomment-995802734
Merged the PR.
Thanks @szlta for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a change in pull request #3748: Hive: ORC vectorization fails when split offsets are considered during split generation
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #3748:
URL: https://github.com/apache/iceberg/pull/3748#discussion_r769570953
##########
File path: mr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
##########
@@ -777,6 +779,42 @@ public void testStatsPopulation() throws Exception {
Assert.assertTrue(stats.startsWith("{\"BASIC_STATS\":\"true\"")); // it's followed by column stats in Hive3
}
+ /**
+ * Tests that vectorized ORC reading code path correctly handles when the same ORC file is split into multiple parts.
+ * Although the split offsets and length will not always include the file tail that contains the metadata, the
+ * vectorized reader needs to make sure to handle the tail reading regardless of the offsets. If this is not done
+ * correctly, the last SELECT query will fail.
+ * @throws Exception - any test error
+ */
+ @Test
+ public void testVectorizedOrcMultipleSplits() throws Exception {
+ assumeTrue(isVectorized && FileFormat.ORC.equals(fileFormat));
+
+ try {
+ // This data will be held by a ~870kB ORC file
+ List<Record> records = TestHelper.generateRandomRecords(HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ 20000, 0L);
+
+ // To support splitting the ORC file, we need to specify the stripe size to a small value. It looks like the min
+ // value is about 220kB, no smaller stripes are written by ORC. Anyway, this setting will produce 4 stripes.
+ shell.getHiveConf().set("orc.stripe.size", "200000");
+
+ testTables.createTable(shell, "targettab", HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ fileFormat, records);
+
+ // Will request 4 splits, separated on the exact stripe boundaries within the ORC file.
+ // (Would request 5 if ORC split generation wouldn't be split (aka stripe) offset aware).
+ shell.getHiveConf().set(InputFormatConfig.SPLIT_SIZE, "200000");
+ List<Object[]> result = shell.executeStatement("SELECT * FROM targettab ORDER BY last_name");
+
+ Assert.assertEquals(20000, result.size());
+ } finally {
+ shell.getHiveConf().unset(InputFormatConfig.SPLIT_SIZE);
Review comment:
`HiveIcebergStorageHandlerTestUtils.init` supposed to take care of the config changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org