You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/25 05:42:53 UTC
[GitHub] [iceberg] wypoon opened a new pull request, #6046: Spark 3.1: Ensure rowStartPosInBatch in ColumnarBatchReader is set correctly
wypoon opened a new pull request, #6046:
URL: https://github.com/apache/iceberg/pull/6046
This is a port of https://github.com/apache/iceberg/pull/6026 to spark/v3.1.
This is not a direct port, as the 3.1 code base lags the 3.3 and 3.2 code base, but it is fairly straightforward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain merged pull request #6046: Spark 3.1: Ensure rowStartPosInBatch in ColumnarBatchReader is set correctly
Posted by GitBox <gi...@apache.org>.
flyrain merged PR #6046:
URL: https://github.com/apache/iceberg/pull/6046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on pull request #6046: Spark 3.1: Ensure rowStartPosInBatch in ColumnarBatchReader is set correctly
Posted by GitBox <gi...@apache.org>.
wypoon commented on PR #6046:
URL: https://github.com/apache/iceberg/pull/6046#issuecomment-1290847165
@flyrain @chenjunjiedada this is the port to 3.1.
cc @hililiwei since you are porting changes to 3.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on a diff in pull request #6046: Spark 3.1: Ensure rowStartPosInBatch in ColumnarBatchReader is set correctly
Posted by GitBox <gi...@apache.org>.
wypoon commented on code in PR #6046:
URL: https://github.com/apache/iceberg/pull/6046#discussion_r1004028589
##########
spark/v3.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java:
##########
@@ -117,16 +131,26 @@ public static void stopMetastoreAndSpark() throws Exception {
spark = null;
}
+ @After
+ @Override
+ public void cleanup() throws IOException {
+ super.cleanup();
+ dropTable("test3");
+ }
+
@Override
protected Table createTable(String name, Schema schema, PartitionSpec spec) {
Table table = catalog.createTable(TableIdentifier.of("default", name), schema);
TableOperations ops = ((BaseTable) table).operations();
TableMetadata meta = ops.current();
ops.commit(meta, meta.upgradeToFormatVersion(2));
+ table
+ .updateProperties()
+ .set(TableProperties.PARQUET_VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .commit();
Review Comment:
`TableProperties.PARQUET_VECTORIZATION_ENABLED_DEFAULT` changed from false to true since this code was written. Thus, before, we only set `TableProperties.PARQUET_VECTORIZATION_ENABLED` explicitly if vectorized is true, but since `TableProperties.PARQUET_VECTORIZATION_ENABLED_DEFAULT` changed to true, all the tests in this class have been running only with vectorized reads and not with non-vectorized reads at all!
This fixes that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org