You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/07/23 22:01:36 UTC
[GitHub] [iceberg] aokolnychyi opened a new pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
aokolnychyi opened a new pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857
This PR adds `DataWriter` builders to our `Avro`, `Parquet`, `ORC` classes. This functionality is needed to introduce a writer factory in subsequent PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi edited a comment on pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#issuecomment-885962685
@kbendick, each introduced data writer builder exposes methods `set` and `setAll` that internally call the corresponding methods in `WriteBuilder`. So you should be able to pass any configuration just like before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi edited a comment on pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi edited a comment on pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#issuecomment-885935538
cc @RussellSpitzer @rdblue @openinx @yyanyy @jackye1995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#discussion_r675910563
##########
File path: core/src/main/java/org/apache/iceberg/avro/Avro.java
##########
@@ -234,6 +235,98 @@ CodecFactory codec() {
}
}
+ public static DataWriteBuilder writeData(OutputFile file) {
+ return new DataWriteBuilder(file);
+ }
+
+ public static class DataWriteBuilder {
+ private final WriteBuilder appenderBuilder;
+ private final String location;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ private DataWriteBuilder(OutputFile file) {
+ this.appenderBuilder = write(file);
+ this.location = file.location();
+ }
+
+ public DataWriteBuilder forTable(Table table) {
+ schema(table.schema());
+ withSpec(table.spec());
+ setAll(table.properties());
+ metricsConfig(MetricsConfig.fromProperties(table.properties()));
+ return this;
+ }
+
+ public DataWriteBuilder schema(org.apache.iceberg.Schema newSchema) {
+ appenderBuilder.schema(newSchema);
+ return this;
+ }
+
+ public DataWriteBuilder set(String property, String value) {
+ appenderBuilder.set(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder setAll(Map<String, String> properties) {
+ appenderBuilder.setAll(properties);
+ return this;
+ }
+
+ public DataWriteBuilder meta(String property, String value) {
+ appenderBuilder.meta(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder overwrite() {
+ return overwrite(true);
+ }
+
+ public DataWriteBuilder overwrite(boolean enabled) {
+ appenderBuilder.overwrite(enabled);
+ return this;
+ }
+
+ public DataWriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
+ appenderBuilder.metricsConfig(newMetricsConfig);
+ return this;
+ }
+
+ public DataWriteBuilder createWriterFunc(Function<Schema, DatumWriter<?>> newCreateWriterFunc) {
+ appenderBuilder.createWriterFunc(newCreateWriterFunc);
+ return this;
+ }
+
+ public DataWriteBuilder withSpec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public DataWriteBuilder withPartition(StructLike newPartition) {
+ this.partition = newPartition;
+ return this;
+ }
+
+ public DataWriteBuilder withKeyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return this;
+ }
+
+ public DataWriteBuilder withSortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return this;
+ }
+
+ public <T> DataWriter<T> buildDataWriter() throws IOException {
Review comment:
Yeah, I was filling the naming in delete writers. I'll switch to `build` here as we only have 1 option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#discussion_r675912448
##########
File path: core/src/main/java/org/apache/iceberg/avro/Avro.java
##########
@@ -234,6 +235,98 @@ CodecFactory codec() {
}
}
+ public static DataWriteBuilder writeData(OutputFile file) {
+ return new DataWriteBuilder(file);
+ }
+
+ public static class DataWriteBuilder {
+ private final WriteBuilder appenderBuilder;
+ private final String location;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ private DataWriteBuilder(OutputFile file) {
+ this.appenderBuilder = write(file);
+ this.location = file.location();
+ }
+
+ public DataWriteBuilder forTable(Table table) {
+ schema(table.schema());
+ withSpec(table.spec());
+ setAll(table.properties());
+ metricsConfig(MetricsConfig.fromProperties(table.properties()));
+ return this;
+ }
+
+ public DataWriteBuilder schema(org.apache.iceberg.Schema newSchema) {
+ appenderBuilder.schema(newSchema);
+ return this;
+ }
+
+ public DataWriteBuilder set(String property, String value) {
+ appenderBuilder.set(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder setAll(Map<String, String> properties) {
+ appenderBuilder.setAll(properties);
+ return this;
+ }
+
+ public DataWriteBuilder meta(String property, String value) {
+ appenderBuilder.meta(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder overwrite() {
+ return overwrite(true);
+ }
+
+ public DataWriteBuilder overwrite(boolean enabled) {
+ appenderBuilder.overwrite(enabled);
+ return this;
+ }
+
+ public DataWriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
+ appenderBuilder.metricsConfig(newMetricsConfig);
+ return this;
+ }
+
+ public DataWriteBuilder createWriterFunc(Function<Schema, DatumWriter<?>> newCreateWriterFunc) {
+ appenderBuilder.createWriterFunc(newCreateWriterFunc);
+ return this;
+ }
+
+ public DataWriteBuilder withSpec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public DataWriteBuilder withPartition(StructLike newPartition) {
+ this.partition = newPartition;
+ return this;
+ }
+
+ public DataWriteBuilder withKeyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return this;
+ }
+
+ public DataWriteBuilder withSortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return this;
+ }
+
+ public <T> DataWriter<T> buildDataWriter() throws IOException {
Review comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi merged pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi merged pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#discussion_r675910563
##########
File path: core/src/main/java/org/apache/iceberg/avro/Avro.java
##########
@@ -234,6 +235,98 @@ CodecFactory codec() {
}
}
+ public static DataWriteBuilder writeData(OutputFile file) {
+ return new DataWriteBuilder(file);
+ }
+
+ public static class DataWriteBuilder {
+ private final WriteBuilder appenderBuilder;
+ private final String location;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ private DataWriteBuilder(OutputFile file) {
+ this.appenderBuilder = write(file);
+ this.location = file.location();
+ }
+
+ public DataWriteBuilder forTable(Table table) {
+ schema(table.schema());
+ withSpec(table.spec());
+ setAll(table.properties());
+ metricsConfig(MetricsConfig.fromProperties(table.properties()));
+ return this;
+ }
+
+ public DataWriteBuilder schema(org.apache.iceberg.Schema newSchema) {
+ appenderBuilder.schema(newSchema);
+ return this;
+ }
+
+ public DataWriteBuilder set(String property, String value) {
+ appenderBuilder.set(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder setAll(Map<String, String> properties) {
+ appenderBuilder.setAll(properties);
+ return this;
+ }
+
+ public DataWriteBuilder meta(String property, String value) {
+ appenderBuilder.meta(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder overwrite() {
+ return overwrite(true);
+ }
+
+ public DataWriteBuilder overwrite(boolean enabled) {
+ appenderBuilder.overwrite(enabled);
+ return this;
+ }
+
+ public DataWriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
+ appenderBuilder.metricsConfig(newMetricsConfig);
+ return this;
+ }
+
+ public DataWriteBuilder createWriterFunc(Function<Schema, DatumWriter<?>> newCreateWriterFunc) {
+ appenderBuilder.createWriterFunc(newCreateWriterFunc);
+ return this;
+ }
+
+ public DataWriteBuilder withSpec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public DataWriteBuilder withPartition(StructLike newPartition) {
+ this.partition = newPartition;
+ return this;
+ }
+
+ public DataWriteBuilder withKeyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return this;
+ }
+
+ public DataWriteBuilder withSortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return this;
+ }
+
+ public <T> DataWriter<T> buildDataWriter() throws IOException {
Review comment:
Yeah, I was following the naming in delete writers. I'll switch to `build` here as we only have 1 option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#issuecomment-885962685
@kbendick, each introduced data writer builder exposes methods `set` and `setAll` that internally call methods the corresponding methods in `WriteBuilder`. So you should be able to pass any configuration just like before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#discussion_r675906351
##########
File path: core/src/main/java/org/apache/iceberg/avro/Avro.java
##########
@@ -234,6 +235,98 @@ CodecFactory codec() {
}
}
+ public static DataWriteBuilder writeData(OutputFile file) {
+ return new DataWriteBuilder(file);
+ }
+
+ public static class DataWriteBuilder {
+ private final WriteBuilder appenderBuilder;
+ private final String location;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ private DataWriteBuilder(OutputFile file) {
+ this.appenderBuilder = write(file);
+ this.location = file.location();
+ }
+
+ public DataWriteBuilder forTable(Table table) {
+ schema(table.schema());
+ withSpec(table.spec());
+ setAll(table.properties());
+ metricsConfig(MetricsConfig.fromProperties(table.properties()));
+ return this;
+ }
+
+ public DataWriteBuilder schema(org.apache.iceberg.Schema newSchema) {
+ appenderBuilder.schema(newSchema);
+ return this;
+ }
+
+ public DataWriteBuilder set(String property, String value) {
+ appenderBuilder.set(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder setAll(Map<String, String> properties) {
+ appenderBuilder.setAll(properties);
+ return this;
+ }
+
+ public DataWriteBuilder meta(String property, String value) {
+ appenderBuilder.meta(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder overwrite() {
+ return overwrite(true);
+ }
+
+ public DataWriteBuilder overwrite(boolean enabled) {
+ appenderBuilder.overwrite(enabled);
+ return this;
+ }
+
+ public DataWriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
+ appenderBuilder.metricsConfig(newMetricsConfig);
+ return this;
+ }
+
+ public DataWriteBuilder createWriterFunc(Function<Schema, DatumWriter<?>> newCreateWriterFunc) {
+ appenderBuilder.createWriterFunc(newCreateWriterFunc);
+ return this;
+ }
+
+ public DataWriteBuilder withSpec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public DataWriteBuilder withPartition(StructLike newPartition) {
+ this.partition = newPartition;
+ return this;
+ }
+
+ public DataWriteBuilder withKeyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return this;
+ }
+
+ public DataWriteBuilder withSortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return this;
+ }
+
+ public <T> DataWriter<T> buildDataWriter() throws IOException {
Review comment:
I think the intent here was to mirror the `DeleteWriteBuilder` that has methods `buildEqualityWriter` and `buildPositionWriter`. But since there is only one `build` method, should this just be named `build`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#issuecomment-885935538
cc @RussellSpitzer @rdblue @openinx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#discussion_r675886513
##########
File path: core/src/main/java/org/apache/iceberg/avro/Avro.java
##########
@@ -234,6 +235,98 @@ CodecFactory codec() {
}
}
+ public static DataWriteBuilder writeData(OutputFile file) {
+ return new DataWriteBuilder(file);
+ }
+
+ public static class DataWriteBuilder {
+ private final WriteBuilder appenderBuilder;
+ private final String location;
+ private PartitionSpec spec = null;
+ private StructLike partition = null;
+ private EncryptionKeyMetadata keyMetadata = null;
+ private SortOrder sortOrder = null;
+
+ private DataWriteBuilder(OutputFile file) {
+ this.appenderBuilder = write(file);
+ this.location = file.location();
+ }
+
+ public DataWriteBuilder forTable(Table table) {
+ schema(table.schema());
+ withSpec(table.spec());
+ setAll(table.properties());
+ metricsConfig(MetricsConfig.fromProperties(table.properties()));
+ return this;
+ }
+
+ public DataWriteBuilder schema(org.apache.iceberg.Schema newSchema) {
+ appenderBuilder.schema(newSchema);
+ return this;
+ }
+
+ public DataWriteBuilder set(String property, String value) {
+ appenderBuilder.set(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder setAll(Map<String, String> properties) {
+ appenderBuilder.setAll(properties);
+ return this;
+ }
+
+ public DataWriteBuilder meta(String property, String value) {
+ appenderBuilder.meta(property, value);
+ return this;
+ }
+
+ public DataWriteBuilder overwrite() {
+ return overwrite(true);
+ }
+
+ public DataWriteBuilder overwrite(boolean enabled) {
+ appenderBuilder.overwrite(enabled);
+ return this;
+ }
+
+ public DataWriteBuilder metricsConfig(MetricsConfig newMetricsConfig) {
+ appenderBuilder.metricsConfig(newMetricsConfig);
+ return this;
+ }
+
+ public DataWriteBuilder createWriterFunc(Function<Schema, DatumWriter<?>> newCreateWriterFunc) {
+ appenderBuilder.createWriterFunc(newCreateWriterFunc);
+ return this;
+ }
+
+ public DataWriteBuilder withSpec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public DataWriteBuilder withPartition(StructLike newPartition) {
+ this.partition = newPartition;
+ return this;
+ }
+
+ public DataWriteBuilder withKeyMetadata(EncryptionKeyMetadata metadata) {
+ this.keyMetadata = metadata;
+ return this;
+ }
+
+ public DataWriteBuilder withSortOrder(SortOrder newSortOrder) {
+ this.sortOrder = newSortOrder;
+ return this;
+ }
+
+ public <T> DataWriter<T> buildDataWriter() throws IOException {
+ Preconditions.checkArgument(spec != null, "Cannot create data writer without spec");
Review comment:
Even if a table is unpartitioned, we still require a spec as we need to use the correct spec id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #2857: Avro, Parquet, ORC: Add DataWriter builders
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #2857:
URL: https://github.com/apache/iceberg/pull/2857#issuecomment-885979238
Thanks, folks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org