You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/27 19:30:46 UTC
[GitHub] [iceberg] jackye1995 commented on a change in pull request #3810: ORC: Add ORC support for write properties
jackye1995 commented on a change in pull request #3810:
URL: https://github.com/apache/iceberg/pull/3810#discussion_r775616550
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -111,6 +111,12 @@ private TableProperties() {
public static final String DELETE_AVRO_COMPRESSION = "write.delete.avro.compression-codec";
public static final String AVRO_COMPRESSION_DEFAULT = "gzip";
+ public static final String ORC_STRIPE_SIZE_BYTES = "write.orc.stripe-size-bytes";
+ public static final String ORC_STRIPE_SIZE_BYTES_DEFAULT = "67108864"; // 64L * 1024 * 1024
Review comment:
nit: can directly be `long`
##########
File path: site/docs/configuration.md
##########
@@ -41,6 +41,8 @@ Iceberg tables support table properties to configure table behavior, like the de
| write.parquet.compression-codec | gzip | Parquet compression codec |
| write.parquet.compression-level | null | Parquet compression level |
| write.avro.compression-codec | gzip | Avro compression codec |
+| write.orc.stripe-size-bytes | 67108864(64 MB) | Define the default ORC stripe size, in bytes. |
Review comment:
nit: align `|` for each column unless it's too long.
##########
File path: orc/src/main/java/org/apache/iceberg/orc/ORC.java
##########
@@ -511,4 +513,14 @@ static Reader newFileReader(InputFile file, Configuration config) {
}
return newFileReader(file.location(), readerOptions);
}
+
+ private static void parseWriterCustomizedConf(Configuration conf) {
+ conf.set(
+ OrcConf.STRIPE_SIZE.name(),
+ conf.get(TableProperties.ORC_STRIPE_SIZE_BYTES, TableProperties.ORC_STRIPE_SIZE_BYTES_DEFAULT));
+
+ conf.set(OrcConf.BLOCK_SIZE.name(), conf.get(
+ TableProperties.ORC_BLOCK_SIZE_BYTES,
+ TableProperties.ORC_BLOCK_SIZE_BYTES_DEFAULT));
Review comment:
```suggestion
conf.set(
OrcConf.BLOCK_SIZE.name(),
conf.get(TableProperties.ORC_BLOCK_SIZE_BYTES, TableProperties.ORC_BLOCK_SIZE_BYTES_DEFAULT));
```
##########
File path: orc/src/main/java/org/apache/iceberg/orc/ORC.java
##########
@@ -145,6 +146,7 @@ public WriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
public <D> FileAppender<D> build() {
Preconditions.checkNotNull(schema, "Schema is required");
+ parseWriterCustomizedConf(conf);
Review comment:
it seems not necessary to have a separated method for this
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -111,6 +111,12 @@ private TableProperties() {
public static final String DELETE_AVRO_COMPRESSION = "write.delete.avro.compression-codec";
public static final String AVRO_COMPRESSION_DEFAULT = "gzip";
+ public static final String ORC_STRIPE_SIZE_BYTES = "write.orc.stripe-size-bytes";
+ public static final String ORC_STRIPE_SIZE_BYTES_DEFAULT = "67108864"; // 64L * 1024 * 1024
+
+ public static final String ORC_BLOCK_SIZE_BYTES = "write.orc.block-size-bytes";
+ public static final String ORC_BLOCK_SIZE_BYTES_DEFAULT = "268435456"; // 256L * 1024 * 1024
Review comment:
nit: can directly be `long`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org