You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/05 04:05:01 UTC
[GitHub] [iceberg] openinx opened a new pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
openinx opened a new pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-643030468
Hi @rdblue , I've removed the flink module with scala 2.11 as you suggested. I've one minor question, when I generate the dependencies.lock file, seems it contains all the `allProcessors` such as:
```
{
"allProcessors": {
"com.github.kevinstern:software-and-algorithms": {
"locked": "1.0",
"transitive": [
"com.google.errorprone:error_prone_check_api"
]
},
"com.github.stephenc.jcip:jcip-annotations": {
"locked": "1.0-1",
"transitive": [
"com.google.errorprone:error_prone_core"
]
},
....
```
That makes the `dependencies.lock` file to be really large. while I saw the file under the other modules such as `iceberg-api` module, it only have the `compile` component. Would this matter ? I followed the guide in your pull requests https://github.com/apache/iceberg/pull/1067.
Any other trick to keep only the `compile` parts ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436040451
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
+
+ @Test
+ public void testConvertFlinkSchemaToIcebergSchema() {
+ TableSchema flinkSchema = TableSchema.builder()
+ .field("id", DataTypes.INT().notNull())
+ .field("name", DataTypes.STRING()) /* optional by default */
+ .field("salary", DataTypes.DOUBLE().notNull())
+ .field("locations", DataTypes.MAP(DataTypes.STRING(),
+ DataTypes.ROW(DataTypes.FIELD("posX", DataTypes.DOUBLE().notNull(), "X field"),
+ DataTypes.FIELD("posY", DataTypes.DOUBLE().notNull(), "Y field"))))
+ .field("strArray", DataTypes.ARRAY(DataTypes.STRING()).nullable())
+ .field("intArray", DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .field("char", DataTypes.CHAR(10).notNull())
+ .field("varchar", DataTypes.VARCHAR(10).notNull())
+ .field("boolean", DataTypes.BOOLEAN().nullable())
+ .field("tinyint", DataTypes.TINYINT())
+ .field("smallint", DataTypes.SMALLINT())
+ .field("bigint", DataTypes.BIGINT())
+ .field("varbinary", DataTypes.VARBINARY(10))
+ .field("binary", DataTypes.BINARY(10))
+ .field("time", DataTypes.TIME())
+ .field("timestamp", DataTypes.TIMESTAMP())
+ .field("date", DataTypes.DATE())
+ .field("decimal", DataTypes.DECIMAL(2, 2))
+ .build();
+
+ Schema actualSchema = FlinkSchemaUtil.convert(flinkSchema);
+ Schema expectedSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.IntegerType.get(), null),
+ Types.NestedField.optional(1, "name", Types.StringType.get(), null),
+ Types.NestedField.required(2, "salary", Types.DoubleType.get(), null),
+ Types.NestedField.optional(3, "locations", Types.MapType.ofOptional(20, 21,
+ Types.StringType.get(),
+ Types.StructType.of(
+ Types.NestedField.required(18, "posX", Types.DoubleType.get(), "X field"),
+ Types.NestedField.required(19, "posY", Types.DoubleType.get(), "Y field")
+ ))),
+ Types.NestedField.optional(4, "strArray", Types.ListType.ofOptional(22, Types.StringType.get())),
+ Types.NestedField.optional(5, "intArray", Types.ListType.ofOptional(23, Types.IntegerType.get())),
+ Types.NestedField.required(6, "char", Types.StringType.get()),
+ Types.NestedField.required(7, "varchar", Types.StringType.get()),
+ Types.NestedField.optional(8, "boolean", Types.BooleanType.get()),
+ Types.NestedField.optional(9, "tinyint", Types.IntegerType.get()),
+ Types.NestedField.optional(10, "smallint", Types.IntegerType.get()),
+ Types.NestedField.optional(11, "bigint", Types.LongType.get()),
+ Types.NestedField.optional(12, "varbinary", Types.BinaryType.get()),
+ Types.NestedField.optional(13, "binary", Types.BinaryType.get()),
+ Types.NestedField.optional(14, "time", Types.TimeType.get()),
+ Types.NestedField.optional(15, "timestamp", Types.TimestampType.withZone()),
+ Types.NestedField.optional(16, "date", Types.DateType.get()),
+ Types.NestedField.optional(17, "decimal", Types.DecimalType.of(2, 2))
+ );
+
+ Assert.assertEquals(expectedSchema.toString(), actualSchema.toString());
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, true, true);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, false, true);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, true, false);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, false, false);
Review comment:
I think validation should be tested separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436031501
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
+ return Types.BinaryType.get();
+ } else if (inner instanceof FloatType) {
+ return Types.FloatType.get();
+ } else if (inner instanceof DoubleType) {
+ return Types.DoubleType.get();
+ } else if (inner instanceof DateType) {
+ return Types.DateType.get();
+ } else if (inner instanceof TimeType) {
+ return Types.TimeType.get();
+ } else if (inner instanceof TimestampType) {
Review comment:
Flink doesn't support `TIMESTAMP WITHOUT TIME ZONE`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-645129975
Ping @rdblue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436852808
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
This is compiling the Iceberg sink against the Flink Scala 2.11 API, right? But Scala isn't binary compatible between minor releases, so Jars compiled for 2.11 may not work with 2.12. How should we avoid problems with binary compatibility?
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
+ return Types.BinaryType.get();
+ } else if (inner instanceof FloatType) {
+ return Types.FloatType.get();
+ } else if (inner instanceof DoubleType) {
+ return Types.DoubleType.get();
+ } else if (inner instanceof DateType) {
+ return Types.DateType.get();
+ } else if (inner instanceof TimeType) {
+ return Types.TimeType.get();
+ } else if (inner instanceof TimestampType) {
Review comment:
It looks like the Flink type that most closely matches is [LocalZonedTimestampType](https://javadoc.io/doc/org.apache.flink/flink-table-common/latest/org/apache/flink/table/types/logical/LocalZonedTimestampType.html) because the Iceberg type does not store the original time zone.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
Review comment:
Looks good now.
##########
File path: api/src/main/java/org/apache/iceberg/Schema.java
##########
@@ -314,4 +315,31 @@ public String toString() {
.map(f -> " " + f)
.collect(Collectors.toList())));
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof Schema)) {
+ return false;
+ }
+
+ Schema that = (Schema) o;
+ if (!Objects.equals(struct.fields(), that.struct.fields())) {
+ return false;
+ }
+
+ if (aliasToId == that.aliasToId) {
+ return true;
+ }
+
+ if (aliasToId == null || that.aliasToId == null) {
+ return false;
+ }
+
+ return aliasToId.equals(that.aliasToId);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(struct.fields(), aliasToId == null ? 0 : aliasToId.hashCode());
+ }
Review comment:
`Schema` doesn't implement `equals` because it isn't clear what schema equality means. Because we track fields by ID, two schemas are equal only if their fields have the same IDs, but most people don't think about schema equality that way and think of a SQL schema, like `id bigint, data string`. To avoid confusion, we don't provide an `equals` method that may have confusing results. Instead, we implement `equals` for structs so you can use `schema.asStruct()` in test assertions.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java
##########
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+
+public class FlinkSchemaUtil {
+
+ private FlinkSchemaUtil() {
+ }
+
+ /**
+ * Convert the flink table schema to apache iceberg schema.
+ */
+ public static Schema convert(TableSchema schema) {
+ Preconditions.checkArgument(schema.toRowDataType() instanceof FieldsDataType, "Should be FieldsDataType");
+
+ FieldsDataType root = (FieldsDataType) schema.toRowDataType();
+ Type converted = FlinkTypeVisitor.visit(root, new FlinkTypeToType(root));
+
+ return new Schema(converted.asNestedType().asStructType().fields());
Review comment:
`asNestedType` is no longer necessary. Now `asStructType` is defined on `Type`, not just `NestedType`.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.table.types.logical.ZonedTimestampType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType fields, List<Type> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == fields;
+
+ List<RowType.RowField> rowFields = ((RowType) fields.getLogicalType()).getFields();
+ Preconditions.checkArgument(rowFields.size() == types.size(), "fields list and types list should have same size.");
+
+ int index = 0;
+ for (int i = 0; i < rowFields.size(); i++) {
+ int id = isRoot ? index : getNextId();
Review comment:
Why have variable `index` when its value is always equal to `i`?
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
I don't see test cases for lists with required elements or lists of structs. Can you add those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r438279601
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
Why use the list's nullability for whether the elements in the list are nullable? It looks like that was correct before.
The [javadoc for `isNullable`](https://javadoc.io/static/org.apache.flink/flink-table-common/1.10.1/org/apache/flink/table/types/logical/LogicalTypeVisitor.html) is "Returns whether a value of this type can be null". That sounds like it describes whether the list can be null or not. The element type has its own nullability that describes whether the elements can be null. I think it was correct before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436036896
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
Review comment:
We typically use a `Pair` in Iceberg code for consistency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436040247
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
Looks like this doesn't test:
* Field comments
* Struct types
* Maps or lists that contain structs
* Maps with non-string keys
Are all of those supported? If so, can you add tests?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-642855351
This looks ready to go once we deduplicate the Flink modules. Whether we depend on 2.11 or 2.12 doesn't matter to me, but I think we should just have one for now, like I said in the thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436032475
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
Review comment:
Minor: We generally prefer `Maps.newLinkedHashMap()` instead of using a specific implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436035765
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
+ }
+ return visitor.fields(fieldsType, fieldResults);
+ } else if (dataType instanceof CollectionDataType) {
+ CollectionDataType collectionType = (CollectionDataType) dataType;
+ return visitor.collection(collectionType,
+ visit(collectionType.getElementDataType(), visitor));
+ } else if (dataType instanceof KeyValueDataType) {
+ KeyValueDataType mapType = (KeyValueDataType) dataType;
+ return visitor.map(mapType,
+ visit(mapType.getKeyDataType(), visitor),
+ visit(mapType.getValueDataType(), visitor));
+ } else if (dataType instanceof AtomicDataType) {
+ AtomicDataType atomic = (AtomicDataType) dataType;
+ return visitor.atomic(atomic);
+ } else {
+ throw new UnsupportedOperationException("Unsupported data type: " + dataType);
+ }
+ }
+
+ public T fields(FieldsDataType dataType, Map<String, Tuple2<String, T>> fieldResults) {
Review comment:
If `fields` is always a row, it seems to me that `row` might be a better name because it is more clear that it is a record/struct/tuple type. It wasn't clear to me that `fields` was the record type, since most other formats make a distinction between a record and its field list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436029382
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.CheckCompatibility;
+import org.apache.iceberg.types.Type;
+
+public class FlinkSchemaUtil {
+
+ private FlinkSchemaUtil() {
+ }
+
+ public static Schema convert(TableSchema flinkSchema) {
+ FieldsDataType root = (FieldsDataType) flinkSchema.toRowDataType();
+ Type converted = FlinkTypeVisitor.visit(root, new FlinkTypeToType(root));
+ return new Schema(converted.asNestedType().asStructType().fields());
+ }
+
+ static void validate(Schema readSchema, Schema writeSchema, boolean checkNullability, boolean checkOrdering) {
+ List<String> errors;
+ if (checkNullability) {
+ errors = CheckCompatibility.writeCompatibilityErrors(readSchema, writeSchema, checkOrdering);
+ } else {
+ errors = CheckCompatibility.typeCompatibilityErrors(readSchema, writeSchema, checkOrdering);
+ }
+ if (!errors.isEmpty()) {
Review comment:
Style: we like to have an empty line after control flow statements like the one above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r437838869
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
I think so, building two `iceberg-flink` jars and deploying them to the repo, so that the downstream users can choose to use the correct `iceberg-flink` jar based on their scala version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436031330
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
Review comment:
Is `BinaryType` a fixed-length binary? If so, it should be `FixedType` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r437104076
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.flink.table.types.logical.ZonedTimestampType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType fields, List<Type> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == fields;
+
+ List<RowType.RowField> rowFields = ((RowType) fields.getLogicalType()).getFields();
+ Preconditions.checkArgument(rowFields.size() == types.size(), "fields list and types list should have same size.");
+
+ int index = 0;
+ for (int i = 0; i < rowFields.size(); i++) {
+ int id = isRoot ? index : getNextId();
Review comment:
OK, the `index` can be removed now (The first version I used a `for (String name : types.keySet())` so defined a `index` variable) .
##########
File path: api/src/main/java/org/apache/iceberg/Schema.java
##########
@@ -314,4 +315,31 @@ public String toString() {
.map(f -> " " + f)
.collect(Collectors.toList())));
}
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof Schema)) {
+ return false;
+ }
+
+ Schema that = (Schema) o;
+ if (!Objects.equals(struct.fields(), that.struct.fields())) {
+ return false;
+ }
+
+ if (aliasToId == that.aliasToId) {
+ return true;
+ }
+
+ if (aliasToId == null || that.aliasToId == null) {
+ return false;
+ }
+
+ return aliasToId.equals(that.aliasToId);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(struct.fields(), aliasToId == null ? 0 : aliasToId.hashCode());
+ }
Review comment:
Got it, thanks.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
+ return Types.BinaryType.get();
+ } else if (inner instanceof FloatType) {
+ return Types.FloatType.get();
+ } else if (inner instanceof DoubleType) {
+ return Types.DoubleType.get();
+ } else if (inner instanceof DateType) {
+ return Types.DateType.get();
+ } else if (inner instanceof TimeType) {
+ return Types.TimeType.get();
+ } else if (inner instanceof TimestampType) {
Review comment:
Read the javadoc again, Yes you're right.
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
Thanks for the reminding. when I provide the unit test for lists, I found a bug in the current patch.
```java
@Override
public Type collection(CollectionDataType collection, Type elementType) {
if (collection.getElementDataType().getLogicalType().isNullable()) {
return Types.ListType.ofOptional(getNextId(), elementType);
} else {
return Types.ListType.ofRequired(getNextId(), elementType);
}
}
```
The `collection.getElementDataType().getLogicalType().isNullable()` should be changed as `collection.getLogicalType().isNullable()`.
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
IMO, we may need to load the corresponding flink Jars (which is depending on scala ) for the configured scala version. For example, if we choose to use the scala 2.12, then the iceberg-flink module should choose to use all the flink-xxx_2.12 jars (should throw error if the flink-xx_2.12 does not exist).
This may need to write few gradlew scripts or methods. I think it could be considered in a separate issue. There're also some other modules such as `iceberg-spark` which should handle this issue too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436030581
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.CheckCompatibility;
+import org.apache.iceberg.types.Type;
+
+public class FlinkSchemaUtil {
+
+ private FlinkSchemaUtil() {
+ }
+
+ public static Schema convert(TableSchema flinkSchema) {
+ FieldsDataType root = (FieldsDataType) flinkSchema.toRowDataType();
+ Type converted = FlinkTypeVisitor.visit(root, new FlinkTypeToType(root));
+ return new Schema(converted.asNestedType().asStructType().fields());
+ }
+
+ static void validate(Schema readSchema, Schema writeSchema, boolean checkNullability, boolean checkOrdering) {
Review comment:
This looks identical to the one in Spark. Can we move this into a helper class so we don't duplicate the code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436576769
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r438508071
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
Review comment:
> Why use the list's nullability for whether the elements in the list are nullable?
I thought the `ListType.ofOptional` says that whether the list is allowed to be nullable. Checked the code, seems it means whether the element in list are allowed to be nullable.
```java
public static class ListType extends NestedType {
public static ListType ofOptional(int elementId, Type elementType) {
Preconditions.checkNotNull(elementType, "Element type cannot be null");
return new ListType(NestedField.optional(elementId, "element", elementType));
}
//....
```
Sorry for this noise, let me check this code again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436428372
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
Review comment:
Seems it was introduced by mistake, will remove this.
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
Review comment:
OK, although we are depending few scala jar from flink modules, we don't need the scala plugin when building the iceberg-flink module because we won't have any scala code.
Will remove this, thanks.
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
+
+ @Test
+ public void testConvertFlinkSchemaToIcebergSchema() {
+ TableSchema flinkSchema = TableSchema.builder()
+ .field("id", DataTypes.INT().notNull())
+ .field("name", DataTypes.STRING()) /* optional by default */
+ .field("salary", DataTypes.DOUBLE().notNull())
+ .field("locations", DataTypes.MAP(DataTypes.STRING(),
+ DataTypes.ROW(DataTypes.FIELD("posX", DataTypes.DOUBLE().notNull(), "X field"),
+ DataTypes.FIELD("posY", DataTypes.DOUBLE().notNull(), "Y field"))))
+ .field("strArray", DataTypes.ARRAY(DataTypes.STRING()).nullable())
+ .field("intArray", DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .field("char", DataTypes.CHAR(10).notNull())
+ .field("varchar", DataTypes.VARCHAR(10).notNull())
+ .field("boolean", DataTypes.BOOLEAN().nullable())
+ .field("tinyint", DataTypes.TINYINT())
+ .field("smallint", DataTypes.SMALLINT())
+ .field("bigint", DataTypes.BIGINT())
+ .field("varbinary", DataTypes.VARBINARY(10))
+ .field("binary", DataTypes.BINARY(10))
+ .field("time", DataTypes.TIME())
+ .field("timestamp", DataTypes.TIMESTAMP())
+ .field("date", DataTypes.DATE())
+ .field("decimal", DataTypes.DECIMAL(2, 2))
+ .build();
+
+ Schema actualSchema = FlinkSchemaUtil.convert(flinkSchema);
+ Schema expectedSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.IntegerType.get(), null),
+ Types.NestedField.optional(1, "name", Types.StringType.get(), null),
+ Types.NestedField.required(2, "salary", Types.DoubleType.get(), null),
+ Types.NestedField.optional(3, "locations", Types.MapType.ofOptional(20, 21,
+ Types.StringType.get(),
+ Types.StructType.of(
+ Types.NestedField.required(18, "posX", Types.DoubleType.get(), "X field"),
+ Types.NestedField.required(19, "posY", Types.DoubleType.get(), "Y field")
+ ))),
+ Types.NestedField.optional(4, "strArray", Types.ListType.ofOptional(22, Types.StringType.get())),
+ Types.NestedField.optional(5, "intArray", Types.ListType.ofOptional(23, Types.IntegerType.get())),
+ Types.NestedField.required(6, "char", Types.StringType.get()),
+ Types.NestedField.required(7, "varchar", Types.StringType.get()),
+ Types.NestedField.optional(8, "boolean", Types.BooleanType.get()),
+ Types.NestedField.optional(9, "tinyint", Types.IntegerType.get()),
+ Types.NestedField.optional(10, "smallint", Types.IntegerType.get()),
+ Types.NestedField.optional(11, "bigint", Types.LongType.get()),
+ Types.NestedField.optional(12, "varbinary", Types.BinaryType.get()),
+ Types.NestedField.optional(13, "binary", Types.BinaryType.get()),
+ Types.NestedField.optional(14, "time", Types.TimeType.get()),
+ Types.NestedField.optional(15, "timestamp", Types.TimestampType.withZone()),
+ Types.NestedField.optional(16, "date", Types.DateType.get()),
+ Types.NestedField.optional(17, "decimal", Types.DecimalType.of(2, 2))
+ );
+
+ Assert.assertEquals(expectedSchema.toString(), actualSchema.toString());
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, true, true);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, false, true);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, true, false);
+ FlinkSchemaUtil.validate(expectedSchema, actualSchema, false, false);
Review comment:
As we discussed above, we will move this `validate` method into the common `TypeUtil`. Let me add unit test in `TestTypeUtil`. Thanks.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
Review comment:
Sounds good.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
Review comment:
Thanks for the reminding, will fix this and provide a unit test to address it. Thanks.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
Review comment:
Yes, it's the only one. Providing a Precondition seems good.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
+ }
+ return visitor.fields(fieldsType, fieldResults);
+ } else if (dataType instanceof CollectionDataType) {
+ CollectionDataType collectionType = (CollectionDataType) dataType;
+ return visitor.collection(collectionType,
+ visit(collectionType.getElementDataType(), visitor));
+ } else if (dataType instanceof KeyValueDataType) {
+ KeyValueDataType mapType = (KeyValueDataType) dataType;
+ return visitor.map(mapType,
+ visit(mapType.getKeyDataType(), visitor),
+ visit(mapType.getValueDataType(), visitor));
+ } else if (dataType instanceof AtomicDataType) {
+ AtomicDataType atomic = (AtomicDataType) dataType;
+ return visitor.atomic(atomic);
+ } else {
+ throw new UnsupportedOperationException("Unsupported data type: " + dataType);
+ }
+ }
+
+ public T fields(FieldsDataType dataType, Map<String, Tuple2<String, T>> fieldResults) {
Review comment:
The name `fields` is matching the `FieldsDataType` , like the `collection` is matching the `CollectionDataType`, `map` is matching the `KeyValueDataType`.
##########
File path: versions.props
##########
@@ -1,5 +1,6 @@
org.slf4j:* = 1.7.25
org.apache.avro:avro = 1.9.2
+org.apache.flink:* = 1.10.0
Review comment:
The flink version can be upgraded to 1.10.1 now.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
+
+ int id = isRoot ? index : getNextId();
+ if (field.getLogicalType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, tuple2.f1, tuple2.f0));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, tuple2.f1, tuple2.f0));
+ }
+ index++;
+ }
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ if (map.getValueDataType().getLogicalType().isNullable()) {
+ return Types.MapType.ofOptional(getNextId(), getNextId(), keyType, valueType);
+ } else {
+ return Types.MapType.ofRequired(getNextId(), getNextId(), keyType, valueType);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:CyclomaticComplexity")
+ @Override
+ public Type atomic(AtomicDataType type) {
+ LogicalType inner = type.getLogicalType();
+ if (inner instanceof VarCharType ||
+ inner instanceof CharType) {
+ return Types.StringType.get();
+ } else if (inner instanceof BooleanType) {
+ return Types.BooleanType.get();
+ } else if (inner instanceof IntType ||
+ inner instanceof SmallIntType ||
+ inner instanceof TinyIntType) {
+ return Types.IntegerType.get();
+ } else if (inner instanceof BigIntType) {
+ return Types.LongType.get();
+ } else if (inner instanceof VarBinaryType ||
+ inner instanceof BinaryType) {
+ return Types.BinaryType.get();
+ } else if (inner instanceof FloatType) {
+ return Types.FloatType.get();
+ } else if (inner instanceof DoubleType) {
+ return Types.DoubleType.get();
+ } else if (inner instanceof DateType) {
+ return Types.DateType.get();
+ } else if (inner instanceof TimeType) {
+ return Types.TimeType.get();
+ } else if (inner instanceof TimestampType) {
Review comment:
It's a bug here which I forget to fix before. we have an issue to address this: https://github.com/generic-datalake/iceberg-pro/issues/30.
Let me address it in this pull request.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
Review comment:
There are two different datatype class in flink type: DataType and LogicalType. The DataType have the following types:
1. AtomicDataType;
2. CollectionDataType;
3. FieldsDataType;
4. KeyValueDataType;
and the logicalType the following kinds: ArrayType/CharType/LongType/MapType etc , it describe more details about the type.
The RowType is a LogicalType, while we defined the visit by accepting DataType, that's why I need two types here: the logical type to get the name & comment, while the DataType to passing the argument.
I think I can write this code more clear, please see the next patch.
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
Yes we have scala 2.12 version. I'm not sure what's the specific problem you mean in 2.11, You have more information ? thanks.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
Review comment:
The reason why I add a `Tuple2/Pair` here because the flink `Type` did not expose the interface to access the `comment`, so I have to maintain it by myself. I got your points about `before` or `after` hooks. In the worst case, the hooks may also accept the `Map<String, Pair<String, Type>>` as an argument. we may could do better ..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-645523276
Looks good now. Thanks, @openinx!
Sorry for the delay getting back to this for a review. I was reviewing a few other big patches in the mean time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r438271764
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
I didn't intend to ask you to add another module to the build. Sorry about not being clear.
In many cases, it's possible to work with both 2.11 and 2.12 with compiled binaries. We have two internal versions of Spark that we do this with, although Spark defines the catalog and table APIs in Java so we have less to worry about. Ideally, we would have one module that works with both and we would have some plan to test that. Otherwise, it's probably a good idea to move this code into Flink as soon as possible because it makes more sense to maintain just source compatibility there.
For now, I think we should consider options for testing this against both 2.11 and 2.12. Maybe we can add a build property to switch between 2.11 and 2.12 and just test both in CI. Until then, I think we should just build for 2.12. 2.11 hasn't been supported since 2017, so it makes sense for new development to target 2.12.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436036606
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
Review comment:
In other visitors, we avoid the need for a `Tuple2` or equivalent by adding a visitor method for a field. That also gives us an opportunity to add before and after callbacks to do things like keep track of the original schema's names. We can always add this later, but you might consider adding it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436028769
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
Review comment:
Is it necessary to have Scala in the Iceberg build? We plan to remove it from the Spark build since it is only used for a couple of helpers. It doesn't look like there is any Scala code in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436039571
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
+
+ @Test
+ public void testConvertFlinkSchemaToIcebergSchema() {
+ TableSchema flinkSchema = TableSchema.builder()
+ .field("id", DataTypes.INT().notNull())
+ .field("name", DataTypes.STRING()) /* optional by default */
+ .field("salary", DataTypes.DOUBLE().notNull())
+ .field("locations", DataTypes.MAP(DataTypes.STRING(),
+ DataTypes.ROW(DataTypes.FIELD("posX", DataTypes.DOUBLE().notNull(), "X field"),
+ DataTypes.FIELD("posY", DataTypes.DOUBLE().notNull(), "Y field"))))
+ .field("strArray", DataTypes.ARRAY(DataTypes.STRING()).nullable())
+ .field("intArray", DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .field("char", DataTypes.CHAR(10).notNull())
+ .field("varchar", DataTypes.VARCHAR(10).notNull())
+ .field("boolean", DataTypes.BOOLEAN().nullable())
+ .field("tinyint", DataTypes.TINYINT())
+ .field("smallint", DataTypes.SMALLINT())
+ .field("bigint", DataTypes.BIGINT())
+ .field("varbinary", DataTypes.VARBINARY(10))
+ .field("binary", DataTypes.BINARY(10))
+ .field("time", DataTypes.TIME())
+ .field("timestamp", DataTypes.TIMESTAMP())
+ .field("date", DataTypes.DATE())
+ .field("decimal", DataTypes.DECIMAL(2, 2))
+ .build();
+
+ Schema actualSchema = FlinkSchemaUtil.convert(flinkSchema);
+ Schema expectedSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.IntegerType.get(), null),
+ Types.NestedField.optional(1, "name", Types.StringType.get(), null),
+ Types.NestedField.required(2, "salary", Types.DoubleType.get(), null),
+ Types.NestedField.optional(3, "locations", Types.MapType.ofOptional(20, 21,
+ Types.StringType.get(),
+ Types.StructType.of(
+ Types.NestedField.required(18, "posX", Types.DoubleType.get(), "X field"),
+ Types.NestedField.required(19, "posY", Types.DoubleType.get(), "Y field")
+ ))),
+ Types.NestedField.optional(4, "strArray", Types.ListType.ofOptional(22, Types.StringType.get())),
+ Types.NestedField.optional(5, "intArray", Types.ListType.ofOptional(23, Types.IntegerType.get())),
+ Types.NestedField.required(6, "char", Types.StringType.get()),
+ Types.NestedField.required(7, "varchar", Types.StringType.get()),
+ Types.NestedField.optional(8, "boolean", Types.BooleanType.get()),
+ Types.NestedField.optional(9, "tinyint", Types.IntegerType.get()),
+ Types.NestedField.optional(10, "smallint", Types.IntegerType.get()),
+ Types.NestedField.optional(11, "bigint", Types.LongType.get()),
+ Types.NestedField.optional(12, "varbinary", Types.BinaryType.get()),
+ Types.NestedField.optional(13, "binary", Types.BinaryType.get()),
+ Types.NestedField.optional(14, "time", Types.TimeType.get()),
+ Types.NestedField.optional(15, "timestamp", Types.TimestampType.withZone()),
+ Types.NestedField.optional(16, "date", Types.DateType.get()),
+ Types.NestedField.optional(17, "decimal", Types.DecimalType.of(2, 2))
+ );
+
+ Assert.assertEquals(expectedSchema.toString(), actualSchema.toString());
Review comment:
Instead of using `toString`, use `asStruct`. Struct implements `equals` so you can use it in tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436039054
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
+
+ @Test
+ public void testConvertFlinkSchemaToIcebergSchema() {
+ TableSchema flinkSchema = TableSchema.builder()
+ .field("id", DataTypes.INT().notNull())
+ .field("name", DataTypes.STRING()) /* optional by default */
+ .field("salary", DataTypes.DOUBLE().notNull())
+ .field("locations", DataTypes.MAP(DataTypes.STRING(),
+ DataTypes.ROW(DataTypes.FIELD("posX", DataTypes.DOUBLE().notNull(), "X field"),
+ DataTypes.FIELD("posY", DataTypes.DOUBLE().notNull(), "Y field"))))
+ .field("strArray", DataTypes.ARRAY(DataTypes.STRING()).nullable())
+ .field("intArray", DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .field("char", DataTypes.CHAR(10).notNull())
+ .field("varchar", DataTypes.VARCHAR(10).notNull())
+ .field("boolean", DataTypes.BOOLEAN().nullable())
+ .field("tinyint", DataTypes.TINYINT())
+ .field("smallint", DataTypes.SMALLINT())
+ .field("bigint", DataTypes.BIGINT())
+ .field("varbinary", DataTypes.VARBINARY(10))
+ .field("binary", DataTypes.BINARY(10))
+ .field("time", DataTypes.TIME())
+ .field("timestamp", DataTypes.TIMESTAMP())
+ .field("date", DataTypes.DATE())
+ .field("decimal", DataTypes.DECIMAL(2, 2))
Review comment:
It would be good to use a different precision and scale so that this validates that they are passed correctly, since both are ints.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436028367
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
Does Flink have a Scala 2.12 version? Will it cause a problem to rely on 2.11?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r437800308
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
+ compile "org.apache.flink:flink-streaming-java_2.11::tests"
Review comment:
My main concern is that if we compile against 2.11, it won't work in 2.12. Then we would need two separate builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436027841
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
+
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+ compile project(':iceberg-arrow')
Review comment:
Is Arrow used in this module?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436576898
##########
File path: flink/src/test/java/org/apache/iceberg/flink/TestFlinkSchemaUtil.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestFlinkSchemaUtil {
+
+ @Test
+ public void testConvertFlinkSchemaToIcebergSchema() {
+ TableSchema flinkSchema = TableSchema.builder()
+ .field("id", DataTypes.INT().notNull())
+ .field("name", DataTypes.STRING()) /* optional by default */
+ .field("salary", DataTypes.DOUBLE().notNull())
+ .field("locations", DataTypes.MAP(DataTypes.STRING(),
+ DataTypes.ROW(DataTypes.FIELD("posX", DataTypes.DOUBLE().notNull(), "X field"),
+ DataTypes.FIELD("posY", DataTypes.DOUBLE().notNull(), "Y field"))))
+ .field("strArray", DataTypes.ARRAY(DataTypes.STRING()).nullable())
+ .field("intArray", DataTypes.ARRAY(DataTypes.INT()).nullable())
+ .field("char", DataTypes.CHAR(10).notNull())
+ .field("varchar", DataTypes.VARCHAR(10).notNull())
+ .field("boolean", DataTypes.BOOLEAN().nullable())
+ .field("tinyint", DataTypes.TINYINT())
+ .field("smallint", DataTypes.SMALLINT())
+ .field("bigint", DataTypes.BIGINT())
+ .field("varbinary", DataTypes.VARBINARY(10))
+ .field("binary", DataTypes.BINARY(10))
+ .field("time", DataTypes.TIME())
+ .field("timestamp", DataTypes.TIMESTAMP())
+ .field("date", DataTypes.DATE())
+ .field("decimal", DataTypes.DECIMAL(2, 2))
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436033182
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
Review comment:
How do we know that the logical type is `RowType`? Is that the only one? We may want to add a Precondition to validate this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JingsongLi commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r441957484
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LocalZonedTimestampType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
Review comment:
A better name is `generateNextId` or just `nextId`. Looks like it is not only "get".
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class FlinkTypeVisitor<T> {
Review comment:
Flink has `LogicalTypeVisitor` and `DataTypeVisitor`, they are very useful for visiting types and you don't need have this `static <T> T visit` method, it is not so elegant.
And for `FieldsDataType`, it not has a good design in 1.9 and 1.10, so in Flink 1.11, it has been refactored to be removed `getFieldDataTypes`.
And I think maybe a `LogicalTypeVisitor` is enough, since we never touch the physical information in the `DataType`s.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LocalZonedTimestampType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType fields, List<Type> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == fields;
+
+ List<RowType.RowField> rowFields = ((RowType) fields.getLogicalType()).getFields();
+ Preconditions.checkArgument(rowFields.size() == types.size(), "fields list and types list should have same size.");
+
+ for (int i = 0; i < rowFields.size(); i++) {
+ int id = isRoot ? i : getNextId();
+
+ RowType.RowField field = rowFields.get(i);
+ String name = field.getName();
+ String comment = field.getDescription().orElse(null);
+
+ if (field.getType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, types.get(i), comment));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, types.get(i), comment));
+ }
+ }
+
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
Review comment:
A `CollectionDataType` may be `MultisetType` too. Maybe iceberg not support it now? Or we can map it to `Map<T, Integer>`.
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LocalZonedTimestampType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType fields, List<Type> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == fields;
+
+ List<RowType.RowField> rowFields = ((RowType) fields.getLogicalType()).getFields();
+ Preconditions.checkArgument(rowFields.size() == types.size(), "fields list and types list should have same size.");
+
+ for (int i = 0; i < rowFields.size(); i++) {
+ int id = isRoot ? i : getNextId();
+
+ RowType.RowField field = rowFields.get(i);
+ String name = field.getName();
+ String comment = field.getDescription().orElse(null);
+
+ if (field.getType().isNullable()) {
+ newFields.add(Types.NestedField.optional(id, name, types.get(i), comment));
+ } else {
+ newFields.add(Types.NestedField.required(id, name, types.get(i), comment));
+ }
+ }
+
+ return Types.StructType.of(newFields);
+ }
+
+ @Override
+ public Type collection(CollectionDataType collection, Type elementType) {
+ if (collection.getElementDataType().getLogicalType().isNullable()) {
+ return Types.ListType.ofOptional(getNextId(), elementType);
+ } else {
+ return Types.ListType.ofRequired(getNextId(), elementType);
+ }
+ }
+
+ @Override
+ public Type map(KeyValueDataType map, Type keyType, Type valueType) {
+ // keys in map are not allowed to be null.
Review comment:
But flink can be, so should we throw unsupported exception here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436028769
##########
File path: build.gradle
##########
@@ -235,6 +235,38 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ apply plugin: 'scala'
Review comment:
Is it necessary to have Scala in the Iceberg build? We plan to remove it from the Spark build since it is only used for a couple of helpers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436034375
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeVisitor.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.RowType;
+
+public class FlinkTypeVisitor<T> {
+
+ static <T> T visit(DataType dataType, FlinkTypeVisitor<T> visitor) {
+ if (dataType instanceof FieldsDataType) {
+ FieldsDataType fieldsType = (FieldsDataType) dataType;
+ Map<String, DataType> fields = fieldsType.getFieldDataTypes();
+ Map<String, Tuple2<String, T>> fieldResults = new LinkedHashMap<>();
+ // Make sure that we're traversing the fields in the same order as constructing the schema's fields.
+ RowType rowType = (RowType) dataType.getLogicalType();
+ for (int i = 0; i < fields.size(); i++) {
+ String name = rowType.getFieldNames().get(i);
+ String comment = rowType.getFields().get(i).getDescription().orElse(null);
+ fieldResults.put(name, Tuple2.of(comment, visit(fields.get(name), visitor)));
Review comment:
What is returned by `rowType.getFields()`? Why is that used to get the comment, but `fields.get(name)` is used for the type? I would expect this to use one or the other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436029382
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkSchemaUtil.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import org.apache.flink.table.api.TableSchema;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.CheckCompatibility;
+import org.apache.iceberg.types.Type;
+
+public class FlinkSchemaUtil {
+
+ private FlinkSchemaUtil() {
+ }
+
+ public static Schema convert(TableSchema flinkSchema) {
+ FieldsDataType root = (FieldsDataType) flinkSchema.toRowDataType();
+ Type converted = FlinkTypeVisitor.visit(root, new FlinkTypeToType(root));
+ return new Schema(converted.asNestedType().asStructType().fields());
+ }
+
+ static void validate(Schema readSchema, Schema writeSchema, boolean checkNullability, boolean checkOrdering) {
+ List<String> errors;
+ if (checkNullability) {
+ errors = CheckCompatibility.writeCompatibilityErrors(readSchema, writeSchema, checkOrdering);
+ } else {
+ errors = CheckCompatibility.typeCompatibilityErrors(readSchema, writeSchema, checkOrdering);
+ }
+ if (!errors.isEmpty()) {
Review comment:
Style: we like to have an empty line after control flow statements like the one above. Looks like we missed this in the original copy of this code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-639624203
Thanks for working on this, @openinx and @waterlx! This is a great start and I think we should be able to get it in soon.
The plan for getting the rest in sounds good to me, and any docs you can share about how Flink will work with Iceberg would be helpful. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r438272539
##########
File path: build.gradle
##########
@@ -235,6 +235,65 @@ project(':iceberg-data') {
}
}
+project(':iceberg-flink') {
+ dependencies {
+ compile project(':iceberg-api')
+ compile project(':iceberg-common')
+ compile project(':iceberg-core')
+ compile project(':iceberg-data')
+ compile project(':iceberg-orc')
+ compile project(':iceberg-parquet')
+
+ compileOnly "org.apache.flink:flink-streaming-java_2.11"
+ compileOnly "org.apache.flink:flink-streaming-java_2.11::tests"
+ compileOnly "org.apache.flink:flink-table-api-java-bridge_2.11"
+ compileOnly "org.apache.flink:flink-table-planner-blink_2.11"
+ compileOnly "org.apache.flink:flink-table-planner_2.11"
+ compileOnly "org.apache.hadoop:hadoop-hdfs"
+ compileOnly "org.apache.hadoop:hadoop-common"
+ compileOnly("org.apache.hadoop:hadoop-minicluster") {
+ exclude group: 'org.apache.avro', module: 'avro'
+ }
+
+ testCompile "org.apache.flink:flink-core"
+ testCompile "org.apache.flink:flink-runtime_2.11"
+ testCompile "org.apache.flink:flink-test-utils-junit"
+ testCompile("org.apache.flink:flink-test-utils_2.11") {
+ exclude group: "org.apache.curator", module: 'curator-test'
+ }
+ }
+}
+
+project(':iceberg-flink_2.12') {
Review comment:
There is no source code in flink_2.12, so this module is actually empty. Like I said in the other comment, let's have one 2.12 module for now to get this PR in. Then we can figure out how to test compatibility with 2.11 later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#discussion_r436037647
##########
File path: flink/src/main/java/org/apache/iceberg/flink/FlinkTypeToType.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.flink;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.table.types.AtomicDataType;
+import org.apache.flink.table.types.CollectionDataType;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.FieldsDataType;
+import org.apache.flink.table.types.KeyValueDataType;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.BinaryType;
+import org.apache.flink.table.types.logical.BooleanType;
+import org.apache.flink.table.types.logical.CharType;
+import org.apache.flink.table.types.logical.DateType;
+import org.apache.flink.table.types.logical.DecimalType;
+import org.apache.flink.table.types.logical.DoubleType;
+import org.apache.flink.table.types.logical.FloatType;
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.SmallIntType;
+import org.apache.flink.table.types.logical.TimeType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.TinyIntType;
+import org.apache.flink.table.types.logical.VarBinaryType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class FlinkTypeToType extends FlinkTypeVisitor<Type> {
+ private final FieldsDataType root;
+ private int nextId = 0;
+
+ FlinkTypeToType(FieldsDataType root) {
+ this.root = root;
+ // the root struct's fields use the first ids
+ this.nextId = root.getFieldDataTypes().size();
+ }
+
+ private int getNextId() {
+ int next = nextId;
+ nextId += 1;
+ return next;
+ }
+
+ @Override
+ public Type fields(FieldsDataType dataType, Map<String, Tuple2<String, Type>> types) {
+ List<Types.NestedField> newFields = Lists.newArrayListWithExpectedSize(types.size());
+ boolean isRoot = root == dataType;
+
+ Map<String, DataType> fieldsMap = dataType.getFieldDataTypes();
+ int index = 0;
+ for (String name : types.keySet()) {
+ assert fieldsMap.containsKey(name);
+ DataType field = fieldsMap.get(name);
+ Tuple2<String, Type> tuple2 = types.get(name);
Review comment:
Using a name like `commentAndType` would help make it clear what's happening when the values are used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on pull request #1096: Add flink data type convertor to convert data types between iceberg data type and flink data type.
Posted by GitBox <gi...@apache.org>.
openinx commented on pull request #1096:
URL: https://github.com/apache/iceberg/pull/1096#issuecomment-643859930
Ping @rdblue , any other concern about this pull request ? If no more concern, please help to merge this patch so that I could PR the following patches, Thanks in advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org