You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:41:05 UTC
[GitHub] [iceberg] cmathiesen opened a new pull request #1103: Add IcebergSerDe
cmathiesen opened a new pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103
Hello! This is part 1 of our series of PR's to add in the mapred InputFormat to support reading tables from Hive. This was initially meant to only include the `IcebergSerDe` but we had to add a few more classes to get it working properly.
@rdblue @massdosage @teabot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r449237263
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestTableResolver.java
##########
@@ -111,4 +111,21 @@ public void resolveTableFromPropertiesDefault() throws IOException {
Assert.assertEquals(tableLocation.getAbsolutePath(), table.location());
}
+ @Test(expected = UnsupportedOperationException.class)
+ public void resolveTableFromConfigurationHiveCatalog() throws IOException {
+ Configuration conf = new Configuration();
+ conf.set(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HIVE_CATALOG);
+ conf.set(InputFormatConfig.TABLE_NAME, "table_a");
+
+ TableResolver.resolveTableFromConfiguration(conf);
+ }
+
+ @Test(expected = NullPointerException.class)
Review comment:
I agree. I didn't want to block progress on Hive just for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] guilload commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
guilload commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439600366
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
Still work in progress, things missing are mostly unit tests, but this is what it'll look like:
https://github.com/guilload/iceberg/commit/3bffa7c87bf6352d5f7832edac383310aaf1df08
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438314390
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
Does this need to be removed from the `InputFormat` class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440898759
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
+
+ private SystemTableUtil() {}
+
+ protected static Schema schemaWithVirtualColumn(Schema schema, String columnName) {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, columnName, Types.LongType.get()));
+ return new Schema(columns);
+ }
+
+ protected static Record recordWithVirtualColumn(Record record, long snapshotId, Schema oldSchema,
+ String columnName) {
+ Schema newSchema = schemaWithVirtualColumn(oldSchema, columnName);
+ Record newRecord = GenericRecord.create(newSchema);
+ for (Types.NestedField field : oldSchema.columns()) {
+ newRecord.setField(field.name(), record.getField(field.name()));
+ }
+ newRecord.setField(columnName, snapshotId);
+ return newRecord;
+ }
+
+ protected static String getVirtualColumnName(Configuration conf) {
+ String virtualColumnName = conf.get(VIRTUAL_COLUMN_NAME);
+ if (virtualColumnName == null) {
+ return DEFAULT_SNAPSHOT_ID_COLUMN_NAME;
+ } else {
+ return virtualColumnName;
+ }
+ }
+
+ protected static String getVirtualColumnName(Properties properties) {
Review comment:
The configuration contains all the configs we set in HiveConf and possible hadoop conf as well.
The properties are a merged result of Hive table and partition properties. We can see how Hive uses these as part of the initialize method of `AbstractSerde`
```
public void initialize(Configuration configuration, Properties tableProperties,
Properties partitionProperties) throws SerDeException {
initialize(configuration,
SerDeUtils.createOverlayedProperties(tableProperties, partitionProperties));
}```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438286558
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
Review comment:
Nit: no need for a blank line here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439507250
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
If we are moving code into its own top-level class, then we need to delete the code that is moving from its original location so we don't drop changes that are introduced in the mean time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] guilload commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
guilload commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-646875440
Yes, please! Github won't let me open a PR from my repo to yours or fork your repo either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440324595
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
Seems like this class is shared by both mrv2 and mrv1 apis, similar to InputFormatConfig. We should have a consistent package name there..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438289177
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
+
+ public void setRecord(Record record) {
+ this.record = record;
+ }
+
+ public Record getRecord() {
Review comment:
For getters, we omit `get` because it doesn't add any value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440485131
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME,
+ conf.get(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES)); //Default to HadoopTables
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES);
+ String tableLocation = properties.getProperty(InputFormatConfig.TABLE_LOCATION);
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ Preconditions.checkNotNull(tableLocation, "Table location is not set.");
+ Preconditions.checkNotNull(tableName, "Table name is not set.");
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
Review comment:
I think this is a good idea, especially if it supports all metadata tables without additional work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439467229
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
"There's actually no need to return anything other the Record object."
@guilload are you sure about this? Ultimately this all ends up being used by the old Map Reduce V1 API where serialization required the usage of Writables. Have you managed to get an InputFormat working end to end in an actual Hive 2.3.x client with a type that isn't a Writable? I see that the InputFormats etc. for Delta and Hudi both seem to use `ArrayWritable`and have Writables scattered over the code which I'm assuming is for this reason. I'd be happy to remove the Writable I'm just not sure it would actually work when run on a cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439464913
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
Review comment:
We discussed this with @rdblue and he didn't want to introduce another sub-package at this stage so we agreed to keep it all here. I don't mind either way to be honest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439074192
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
Review comment:
If this is to follow an established pattern in Hive, I'm happy with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438317771
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
I think the warehouse path should be passed separately, possibly as table properties but maybe just defaulting to the Hive warehouse path. It doesn't make sense to me to pass a full URI and then use HadoopCatalog instead of HadoopTables, which handles tables at a specific path. There is no benefit to HadoopCatalog if you have the full table location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] guilload commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
guilload commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439646157
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
> > Ultimately this all ends up being used by the old Map Reduce V1 API where serialization required the usage of Writables
>
> I think the suggestion was to return `Record` from `deserialize`, not to use a non-writable. After `deserialize`, I don't think there is a requirement for anything to be a `Writable`.
Exactly. Until getting to `deserialize` the object has to be a `Writable` but after that point, it can be anything really, as long as the right object inspector is provided. I have already successfully tested this approach on local branch where I built a simple Hive input format on top of #1104 and implemented my own SerDe with [this object inspector](https://github.com/guilload/iceberg/commit/3bffa7c87bf6352d5f7832edac383310aaf1df08#diff-0819121b2ca0adb7dfc72ceb03dd97b5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] guilload commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
guilload commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439002574
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
There's actually no need to return anything other the `Record` object. Hive relies on the `ObjectInspector` class to retrieve values from arbitrary data structures and convert those values to the classes Hive expects.
You may want to take a look at https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-ObjectInspector and then read the implementation of a few object inspectors in the Hive codebase.
So all we have to do here is return the `Record` object and ensure we have the right object inspectors for the `Record` class and other data types that are not represented in the same fashion in Iceberg vs. Hive.
As a matter of fact, I started working on an object inspector for the `Record` class last year. Why don't I submit a PR on top off this one to get you started?
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
I already have a type visitor somewhere from `Schema` to `ObjectInspector`. I can also submit that one on my PR so you can focus on the remaining things to do.
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
+ IcebergWritable icebergWritable = (IcebergWritable) writable;
+ List<Types.NestedField> fields = icebergWritable.schema().columns();
+
+ if (row == null || row.size() != fields.size()) {
+ row = new ArrayList<Object>(fields.size());
+ } else {
+ row.clear();
+ }
+ for (int i = 0; i < fields.size(); i++) {
+ Object obj = ((IcebergWritable) writable).record().get(i);
+ Type fieldType = fields.get(i).type();
+ if (fieldType.equals(Types.DateType.get())) {
+ row.add(Date.valueOf((LocalDate) obj));
Review comment:
See this is where object inspectors come in handy, they'll take care of the conversion for us. Plus, you can clearly see that this current approach won't work for nested data structures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447350096
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergRecordObjectInspector.java
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.StructField;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+
+public final class IcebergRecordObjectInspector extends StructObjectInspector {
+
+ private static final IcebergRecordObjectInspector EMPTY =
+ new IcebergRecordObjectInspector(Types.StructType.of(), Collections.emptyList());
+
+ private final List<IcebergRecordStructField> structFields;
+
+ public IcebergRecordObjectInspector(Types.StructType structType, List<ObjectInspector> objectInspectors) {
+ Preconditions.checkArgument(structType.fields().size() == objectInspectors.size());
+
+ this.structFields = Lists.newArrayListWithExpectedSize(structType.fields().size());
+
+ int position = 0;
+
+ for (Types.NestedField field : structType.fields()) {
+ ObjectInspector oi = objectInspectors.get(position);
+ IcebergRecordStructField structField = new IcebergRecordStructField(field, oi, position);
+ structFields.add(structField);
+ position++;
+ }
+ }
+
+ public static IcebergRecordObjectInspector empty() {
+ return EMPTY;
+ }
+
+ @Override
+ public List<? extends StructField> getAllStructFieldRefs() {
+ return structFields;
+ }
+
+ @Override
+ public StructField getStructFieldRef(String name) {
+ return ObjectInspectorUtils.getStandardStructFieldRef(name, structFields);
+ }
+
+ @Override
+ public Object getStructFieldData(Object o, StructField structField) {
+ return ((Record) o).get(((IcebergRecordStructField) structField).position());
+ }
+
+ @Override
+ public List<Object> getStructFieldsDataAsList(Object o) {
+ Record record = (Record) o;
+ return structFields
+ .stream()
+ .map(f -> record.get(f.position()))
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public String getTypeName() {
+ return ObjectInspectorUtils.getStandardStructTypeName(this);
+ }
+
+ @Override
+ public Category getCategory() {
+ return Category.STRUCT;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ IcebergRecordObjectInspector that = (IcebergRecordObjectInspector) o;
+ return structFields.equals(that.structFields);
+ }
+
+ @Override
+ public int hashCode() {
+ return structFields.hashCode();
+ }
+
+ private static class IcebergRecordStructField implements StructField {
+
+ private final Types.NestedField field;
+ private final ObjectInspector oi;
+ private final int position;
+
+ IcebergRecordStructField(Types.NestedField field, ObjectInspector oi, int position) {
+ this.field = field;
+ this.oi = oi;
+ this.position = position; // position in the record
+ }
+
+ @Override
+ public String getFieldName() {
+ return field.name();
+ }
+
+ @Override
+ public ObjectInspector getFieldObjectInspector() {
+ return oi;
+ }
+
+ @Override
+ public int getFieldID() {
+ return field.fieldId();
+ }
+
+ @Override
+ public String getFieldComment() {
+ return field.doc();
+ }
+
+ int position() {
+ return position;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ IcebergRecordStructField that = (IcebergRecordStructField) o;
+ return field.equals(that.field) && oi.equals(that.oi);
+ }
+
+ @Override
+ public int hashCode() {
+ return 31 * field.hashCode() + oi.hashCode();
Review comment:
We typically prefer `Objects.hash` to this older pattern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448516088
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
@rdblue , @guilload, @massdosage if there are many issues, open items, does it make sense to create a milestone with all the open tickets? So that it can be worked on in parallel by us and we don't duplicate effort or step on each other's toes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448814275
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestTableResolver.java
##########
@@ -111,4 +111,21 @@ public void resolveTableFromPropertiesDefault() throws IOException {
Assert.assertEquals(tableLocation.getAbsolutePath(), table.location());
}
+ @Test(expected = UnsupportedOperationException.class)
+ public void resolveTableFromConfigurationHiveCatalog() throws IOException {
+ Configuration conf = new Configuration();
+ conf.set(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HIVE_CATALOG);
+ conf.set(InputFormatConfig.TABLE_NAME, "table_a");
+
+ TableResolver.resolveTableFromConfiguration(conf);
+ }
+
+ @Test(expected = NullPointerException.class)
Review comment:
Since the above is almost certainly going to be refactored when we merge more logic between the two InputFormats we can tackle this then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440904139
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
+ case LIST:
+ ListTypeInfo ati = (ListTypeInfo) typeInfo;
+ return ObjectInspectorFactory
+ .getStandardListObjectInspector(createObjectInspectorWorker(ati.getListElementTypeInfo()));
+ case MAP:
+ MapTypeInfo mti = (MapTypeInfo) typeInfo;
+ return ObjectInspectorFactory.getStandardMapObjectInspector(
+ createObjectInspectorWorker(mti.getMapKeyTypeInfo()),
+ createObjectInspectorWorker(mti.getMapValueTypeInfo()));
+ case STRUCT:
+ StructTypeInfo sti = (StructTypeInfo) typeInfo;
+ List<ObjectInspector> ois = new ArrayList<>(sti.getAllStructFieldTypeInfos().size());
+ for (TypeInfo structTypeInfos : sti.getAllStructFieldTypeInfos()) {
+ ois.add(createObjectInspectorWorker(structTypeInfos));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(sti.getAllStructFieldNames(), ois);
+ default:
+ throw new SerDeException("Couldn't create Object Inspector for category: '" + typeCategory + "'");
+ }
+ }
+
+ protected List<String> setColumnNames(Schema schema) {
Review comment:
nit: maybe just rename this to `columnNames`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448501109
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
I thought the convention was that the package name needed to match the subproject name and this isn't in the `hive` package but maybe that's not the case? Alternatively they could go in `org.apache.iceberg.mr.hive`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448541324
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
`mr.hive` sounds good to me.
I don't think we need to worry too much about this kind of refactor right now. We expect it to change rapidly as we build. We'll include a note in any release about how it is experimental and subject to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440904139
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
+ case LIST:
+ ListTypeInfo ati = (ListTypeInfo) typeInfo;
+ return ObjectInspectorFactory
+ .getStandardListObjectInspector(createObjectInspectorWorker(ati.getListElementTypeInfo()));
+ case MAP:
+ MapTypeInfo mti = (MapTypeInfo) typeInfo;
+ return ObjectInspectorFactory.getStandardMapObjectInspector(
+ createObjectInspectorWorker(mti.getMapKeyTypeInfo()),
+ createObjectInspectorWorker(mti.getMapValueTypeInfo()));
+ case STRUCT:
+ StructTypeInfo sti = (StructTypeInfo) typeInfo;
+ List<ObjectInspector> ois = new ArrayList<>(sti.getAllStructFieldTypeInfos().size());
+ for (TypeInfo structTypeInfos : sti.getAllStructFieldTypeInfos()) {
+ ois.add(createObjectInspectorWorker(structTypeInfos));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(sti.getAllStructFieldNames(), ois);
+ default:
+ throw new SerDeException("Couldn't create Object Inspector for category: '" + typeCategory + "'");
+ }
+ }
+
+ protected List<String> setColumnNames(Schema schema) {
Review comment:
nit: maybe just rename this to `columnNames`. Also why is this `protected` , seems like it can static like its counterpart in `IcebergSchemaToTypeInfo`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441644030
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
In this PR it's only being used by the mrv1 API which is why it's in the `mapred` package. When we look at #933 after this we can discuss how much code is shared and then potentially move this up a package level and have it contain shared code etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438728593
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
Review comment:
Yeah, it supports all primitive types as keys I believe - I've added an extra test case using an Integer key type, but should I add a larger range of tests like this one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439073797
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
Review comment:
No, one non-string test case is good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448516088
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
@rdblue , @guilload, @massdosage if there are many issues, open items does it make sense to create a milestone with all the open tickets? So that it can be worked on in parallel by us and we don't duplicate effort or step on each other's toes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448504948
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
I prefer `org.apache.iceberg.mr.hive`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447364952
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergDateObjectInspector.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.sql.Date;
+import java.time.LocalDate;
+import org.apache.hadoop.hive.serde2.io.DateWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.DateObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+
+public final class IcebergDateObjectInspector extends IcebergPrimitiveObjectInspector implements DateObjectInspector {
+
+ private static final IcebergDateObjectInspector INSTANCE = new IcebergDateObjectInspector();
+
+ public static IcebergDateObjectInspector get() {
+ return INSTANCE;
+ }
+
+ private IcebergDateObjectInspector() {
+ super(TypeInfoFactory.dateTypeInfo);
+ }
+
+ @Override
+ public Date getPrimitiveJavaObject(Object o) {
+ return o == null ? null : Date.valueOf((LocalDate) o);
+ }
+
+ @Override
+ public DateWritable getPrimitiveWritableObject(Object o) {
+ Date date = getPrimitiveJavaObject(o);
+ return date == null ? null : new DateWritable(date);
Review comment:
Instead of converting to `Date` and then wrapping with `DateWritable`, could we use the `DateWritable` constructor that accepts an integer? That would be more direct and we could convert using `DateTimeUtil.daysFromDate(localDate)` that we use elsewhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439511225
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
Review comment:
As long as the Hive dependencies are provided/compileOnly and we can still use the MR models, I don't see much harm in keeping them in one module.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439658065
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME,
+ conf.get(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES)); //Default to HadoopTables
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES);
+ String tableLocation = properties.getProperty(InputFormatConfig.TABLE_LOCATION);
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ Preconditions.checkNotNull(tableLocation, "Table location is not set.");
+ Preconditions.checkNotNull(tableName, "Table name is not set.");
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
Review comment:
According to https://github.com/ExpediaGroup/hiveberg/blob/master/README.md#time-travel-and-system-tables it seems like to access a snapshot metadata table, we need to create a new table which ends in `__snapshots`. Instead of that, can we rather let the user create a table with any name but put the metadata table name as part of the location instead.
```
CREATE TABLE source_db.table_a_snapshot_metadata_table
STORED BY 'com.expediagroup.hiveberg.IcebergStorageHandler'
LOCATION 'path_to_original_data_table#snapshots'
TBLPROPERTIES ('iceberg.catalog'='hadoop.catalog')
```
That way,
1) You don't need to reserve a table suffix
2) You don't need flags like `iceberg.snapshots.table=false`
3) TableResolver code is also simplified as it does not need to understand the syntax for metadata tables in different catalogs. e.g. HadoopTables uses `#snapshots`, HiveCatalog uses `.snapshots`. It simply has to pass the location as-is to the catalog.
4) It will also be able to handle other metadata tables, like partitions, manifest, entries, etc. without any additional handling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438288977
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
+
+ public void setRecord(Record record) {
Review comment:
We typically use the verb `wrap` for this pattern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r446165329
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
Above changes have now been merged into this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447344934
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.mr.mapred.serde.objectinspector.IcebergObjectInspector;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private ObjectInspector inspector;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ final Table table;
+
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
Review comment:
I didn't realize Java added an `UncheckedIOException` in 8. We have one that is `RuntimeIOException`. We should probably convert Iceberg over to using the standard Java one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448813803
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
I think the only missing necessary follow up here already has a PR at #1157 so I'm not sure we need to do this? We're now co-ordinating with @guilload on the next steps for the `mapred` InputFormat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439517042
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
> > Ultimately this all ends up being used by the old Map Reduce V1 API where serialization required the usage of Writables
>
> I think the suggestion was to return `Record` from `deserialize`, not to use a non-writable. After `deserialize`, I don't think there is a requirement for anything to be a `Writable`.
We can certainly try, we should probably do it in Hiveberg first as we can then run a full HiveRunner test over it to make sure there aren't any surprises in the internals of Hive or MR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438293053
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
+
+ private SystemTableUtil() {}
+
+ protected static Schema schemaWithVirtualColumn(Schema schema, String columnName) {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, columnName, Types.LongType.get()));
+ return new Schema(columns);
+ }
+
+ protected static Record recordWithVirtualColumn(Record record, long snapshotId, Schema oldSchema,
+ String columnName) {
+ Schema newSchema = schemaWithVirtualColumn(oldSchema, columnName);
+ Record newRecord = GenericRecord.create(newSchema);
+ for (Types.NestedField field : oldSchema.columns()) {
Review comment:
The field positions match, right? If so, then this could just iterate through positions in the original record instead of names. That would avoid a hashmap lookup of the position for both get and set.
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
+
+ private SystemTableUtil() {}
+
+ protected static Schema schemaWithVirtualColumn(Schema schema, String columnName) {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, columnName, Types.LongType.get()));
+ return new Schema(columns);
+ }
+
+ protected static Record recordWithVirtualColumn(Record record, long snapshotId, Schema oldSchema,
+ String columnName) {
+ Schema newSchema = schemaWithVirtualColumn(oldSchema, columnName);
+ Record newRecord = GenericRecord.create(newSchema);
+ for (Types.NestedField field : oldSchema.columns()) {
+ newRecord.setField(field.name(), record.getField(field.name()));
+ }
+ newRecord.setField(columnName, snapshotId);
+ return newRecord;
+ }
+
+ protected static String getVirtualColumnName(Configuration conf) {
+ String virtualColumnName = conf.get(VIRTUAL_COLUMN_NAME);
+ if (virtualColumnName == null) {
+ return DEFAULT_SNAPSHOT_ID_COLUMN_NAME;
+ } else {
+ return virtualColumnName;
+ }
+ }
+
+ protected static String getVirtualColumnName(Properties properties) {
Review comment:
When are properties used and when is configuration used? I'm surprised that we need both.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-650172547
All the outstanding comments on this PR have now been resolved. Could we please get another round of reviews to see if this can be merged now? @rdblue @rdsr
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438643554
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
Review comment:
We ended up going with the double underscore option to follow the convention of the inbuilt virtual columns in Hive: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VirtualColumns
I agree it does add a little bit of extra confusion with the slight difference between Iceberg's column name and our choice for the virtual column name, but the argument could go either way. We're happy to go for any option as it'll just be a matter of documenting - do you have a preference?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439073473
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
+ Map<String, String> expected = ImmutableMap.of("foo", "bar");
+ List<Map> data = new ArrayList<>();
+ data.add(expected);
+
+ Record record = TestHelpers.createCustomRecord(schema, data);
+ IcebergWritable writable = new IcebergWritable();
+ writable.setRecord(record);
+ writable.setSchema(schema);
+
+ IcebergSerDe serDe = new IcebergSerDe();
+ List<Object> deserialized = (List<Object>) serDe.deserialize(writable);
+ Map result = (Map) deserialized.get(0);
+
+ assertEquals(expected, result);
+ assertTrue(result.containsKey("foo"));
+ assertTrue(result.containsValue("bar"));
+ }
+
+ @Test
+ public void testDeserializeList() {
Review comment:
And if we take the advice from @guilload and just return the record, then we don't need to test at all! We could convert the whole suite to testing the object inspectors against Iceberg generics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438694487
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
That looks way simpler, I'll get started on that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440146643
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
That looks promising, happy to move that in here when you're done if the others agree. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440909271
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
Review comment:
should we map BYTE, SHORT to Integer here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438289470
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
+
+ public void setRecord(Record record) {
+ this.record = record;
+ }
+
+ public Record getRecord() {
+ return record;
+ }
+
+ public Schema getSchema() {
+ return schema;
+ }
+
+ public void setSchema(Schema schema) {
+ this.schema = schema;
+ }
+
+ @Override
+ public void write(DataOutput dataOutput) throws IOException {
+
Review comment:
If this writable isn't actually writable, then I think this should throw `UnsupportedOperationException` here and in `readFields`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447349868
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergRecordObjectInspector.java
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
+import org.apache.hadoop.hive.serde2.objectinspector.StructField;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+
+public final class IcebergRecordObjectInspector extends StructObjectInspector {
+
+ private static final IcebergRecordObjectInspector EMPTY =
+ new IcebergRecordObjectInspector(Types.StructType.of(), Collections.emptyList());
+
+ private final List<IcebergRecordStructField> structFields;
+
+ public IcebergRecordObjectInspector(Types.StructType structType, List<ObjectInspector> objectInspectors) {
+ Preconditions.checkArgument(structType.fields().size() == objectInspectors.size());
+
+ this.structFields = Lists.newArrayListWithExpectedSize(structType.fields().size());
+
+ int position = 0;
+
+ for (Types.NestedField field : structType.fields()) {
+ ObjectInspector oi = objectInspectors.get(position);
+ IcebergRecordStructField structField = new IcebergRecordStructField(field, oi, position);
+ structFields.add(structField);
+ position++;
+ }
+ }
+
+ public static IcebergRecordObjectInspector empty() {
+ return EMPTY;
+ }
+
+ @Override
+ public List<? extends StructField> getAllStructFieldRefs() {
+ return structFields;
+ }
+
+ @Override
+ public StructField getStructFieldRef(String name) {
+ return ObjectInspectorUtils.getStandardStructFieldRef(name, structFields);
+ }
+
+ @Override
+ public Object getStructFieldData(Object o, StructField structField) {
+ return ((Record) o).get(((IcebergRecordStructField) structField).position());
+ }
+
+ @Override
+ public List<Object> getStructFieldsDataAsList(Object o) {
+ Record record = (Record) o;
+ return structFields
+ .stream()
+ .map(f -> record.get(f.position()))
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public String getTypeName() {
+ return ObjectInspectorUtils.getStandardStructTypeName(this);
+ }
+
+ @Override
+ public Category getCategory() {
+ return Category.STRUCT;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ IcebergRecordObjectInspector that = (IcebergRecordObjectInspector) o;
+ return structFields.equals(that.structFields);
+ }
+
+ @Override
+ public int hashCode() {
+ return structFields.hashCode();
+ }
+
+ private static class IcebergRecordStructField implements StructField {
+
+ private final Types.NestedField field;
+ private final ObjectInspector oi;
+ private final int position;
+
+ IcebergRecordStructField(Types.NestedField field, ObjectInspector oi, int position) {
+ this.field = field;
+ this.oi = oi;
+ this.position = position; // position in the record
+ }
+
+ @Override
+ public String getFieldName() {
+ return field.name();
+ }
+
+ @Override
+ public ObjectInspector getFieldObjectInspector() {
+ return oi;
+ }
+
+ @Override
+ public int getFieldID() {
Review comment:
@omalley, is the Iceberg field ID suitable to return as a Hive field ID here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439312426
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
OK, may I suggest we just focus on supporting HadoopTables using the location for now and then add support for HiveCatalog in a future PR after this is merged? Just to keep the scope of this PR down etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r436847180
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
+ if (primitiveTypeToTypeInfo.containsKey(type)) {
+ return (TypeInfo) primitiveTypeToTypeInfo.get(type);
+ }
+ switch (type.typeId()) {
+ case UUID:
+ return TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME);
+ case FIXED:
+ return TypeInfoFactory.getPrimitiveTypeInfo("binary");
+ case TIME:
+ return TypeInfoFactory.getPrimitiveTypeInfo("long");
+ case DECIMAL:
+ Types.DecimalType dec = (Types.DecimalType) type;
+ int scale = dec.scale();
+ int precision = dec.precision();
+ try {
+ HiveDecimalUtils.validateParameter(precision, scale);
+ } catch (Exception e) {
+ //TODO Log that precision / scale isn't valid
Review comment:
Can you add the log message you want now? Or just remove the try/catch as it's not adding anything right now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438288570
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
Review comment:
Since this is public, can you add Javadoc describing its purpose?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441211892
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -517,20 +411,20 @@ private Record withIdentityPartitionColumns(
}
private static Table findTable(Configuration conf) {
- String path = conf.get(TABLE_PATH);
+ String path = conf.get(InputFormatConfig.TABLE_PATH);
Preconditions.checkArgument(path != null, "Table path should not be null");
if (path.contains("/")) {
HadoopTables tables = new HadoopTables(conf);
return tables.load(path);
}
- String catalogFuncClass = conf.get(CATALOG);
+ String catalogFuncClass = conf.get(InputFormatConfig.CATALOG);
if (catalogFuncClass != null) {
Function<Configuration, Catalog> catalogFunc = (Function<Configuration, Catalog>)
- DynConstructors.builder(Function.class)
- .impl(catalogFuncClass)
- .build()
- .newInstance();
+ DynConstructors.builder(Function.class)
+ .impl(catalogFuncClass)
+ .build()
+ .newInstance();
Review comment:
Can you revert the whitespace-only changes in this file? Otherwise this is going to have more conflicts than necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438317000
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
Review comment:
Why does this coerce to a URI and get the path? The full path should be used so I don't see a reason to modify it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438304950
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, SystemTableUtil.getVirtualColumnName(serDeProperties),
+ Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
+ IcebergWritable icebergWritable = (IcebergWritable) writable;
+ List<Types.NestedField> fields = icebergWritable.getSchema().columns();
+ List<Object> row = new ArrayList<>();
Review comment:
Does this need to allocate a new ArrayList every time or can it reuse one?
We try to make the code called for every row in a tight loop (like this method) as small as possible for performance reasons. Ideally, we would be able to reuse this storage, have the list of columns already prepared, and access field values by column position instead of by name. Something like this:
```java
public Object deserialize(Writable writable) {
Record record = ((IcebergWritable) writable).record();
for (int i = 0; i < recordSize; i += 1) {
reusedArray[i] = record.get(i, Object.class);
}
return reusedArrayAsList;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439291290
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
@guilload that would be great, thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438288656
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
Review comment:
No need to include empty constructors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439072857
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
I think it's safer to either reference it in place or to move it in this commit. Otherwise, changes to the other class may not show up as git commit conflicts and we could accidentally lose them by adding the class in one commit and removing the other copy later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-651476463
I had a few questions, but overall this looks good. The only blocker is how `ByteBuffer` is handled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439508077
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
Yeah, that sounds good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447346661
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergBinaryObjectInspector.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.hadoop.io.BytesWritable;
+
+public final class IcebergBinaryObjectInspector extends IcebergPrimitiveObjectInspector
+ implements BinaryObjectInspector {
+
+ private static final IcebergBinaryObjectInspector INSTANCE = new IcebergBinaryObjectInspector();
+
+ public static IcebergBinaryObjectInspector get() {
+ return INSTANCE;
+ }
+
+ private IcebergBinaryObjectInspector() {
+ super(TypeInfoFactory.binaryTypeInfo);
+ }
+
+ @Override
+ public byte[] getPrimitiveJavaObject(Object o) {
+ return o == null ? null : ((ByteBuffer) o).array();
Review comment:
This isn't correct because it doesn't follow the contract of `ByteBuffer`. Avro will reuse byte buffers, so there is no guarantee that this array is the correct length. In addition, we want to generally follow the `ByteBuffer` contract so that we don't need to worry about whether an optimization later (buffer reuse) will break certain sections of code.
An easy fix is to use `ByteBuffers.toByteArray` here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447348111
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergObjectInspector.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public final class IcebergObjectInspector extends TypeUtil.SchemaVisitor<ObjectInspector> {
+
+ public static ObjectInspector create(@Nullable Schema schema) {
+ if (schema == null) {
+ return IcebergRecordObjectInspector.empty();
+ }
+
+ return TypeUtil.visit(schema, new IcebergObjectInspector());
+ }
+
+ public static ObjectInspector create(Types.NestedField... fields) {
+ return create(new Schema(fields));
+ }
+
+ @Override
+ public ObjectInspector field(Types.NestedField field, ObjectInspector fieldObjectInspector) {
+ return fieldObjectInspector;
+ }
+
+ @Override
+ public ObjectInspector list(Types.ListType listTypeInfo, ObjectInspector listObjectInspector) {
+ return ObjectInspectorFactory.getStandardListObjectInspector(listObjectInspector);
+ }
+
+ @Override
+ public ObjectInspector map(Types.MapType mapType,
+ ObjectInspector keyObjectInspector, ObjectInspector valueObjectInspector) {
+ return ObjectInspectorFactory.getStandardMapObjectInspector(keyObjectInspector, valueObjectInspector);
+ }
+
+ @Override
+ public ObjectInspector primitive(Type.PrimitiveType primitiveType) {
+ final PrimitiveTypeInfo primitiveTypeInfo;
+
+ switch (primitiveType.typeId()) {
+ case BINARY:
+ return IcebergBinaryObjectInspector.get();
+ case BOOLEAN:
+ primitiveTypeInfo = TypeInfoFactory.booleanTypeInfo;
+ break;
+ case DATE:
+ return IcebergDateObjectInspector.get();
+ case DECIMAL:
+ Types.DecimalType type = (Types.DecimalType) primitiveType;
+ return IcebergDecimalObjectInspector.get(type.precision(), type.scale());
+ case DOUBLE:
+ primitiveTypeInfo = TypeInfoFactory.doubleTypeInfo;
+ break;
+ case FLOAT:
+ primitiveTypeInfo = TypeInfoFactory.floatTypeInfo;
+ break;
+ case INTEGER:
+ primitiveTypeInfo = TypeInfoFactory.intTypeInfo;
+ break;
+ case LONG:
+ primitiveTypeInfo = TypeInfoFactory.longTypeInfo;
+ break;
+ case STRING:
+ primitiveTypeInfo = TypeInfoFactory.stringTypeInfo;
+ break;
+ case TIMESTAMP:
+ boolean adjustToUTC = ((Types.TimestampType) primitiveType).shouldAdjustToUTC();
+ return IcebergTimestampObjectInspector.get(adjustToUTC);
+
+ case FIXED:
+ case TIME:
+ case UUID:
+ default:
+ throw new IllegalArgumentException(primitiveType.typeId() + " type is not supported");
Review comment:
Couldn't fixed by read as binary? And UUID as string? And doesn't Hive support time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438289470
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
+
+ public void setRecord(Record record) {
+ this.record = record;
+ }
+
+ public Record getRecord() {
+ return record;
+ }
+
+ public Schema getSchema() {
+ return schema;
+ }
+
+ public void setSchema(Schema schema) {
+ this.schema = schema;
+ }
+
+ @Override
+ public void write(DataOutput dataOutput) throws IOException {
+
Review comment:
If this writable isn't actually writable, then I think this should throw `UnsupportedOperationException`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438288656
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergWritable.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+
+public class IcebergWritable implements Writable {
+
+ private Record record;
+ private Schema schema;
+
+ public IcebergWritable() {}
Review comment:
No need to include empty `public` constructors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shardulm94 commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
shardulm94 commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439080230
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
Instead of using ObjectInspectors, do you think it's worth building a ReaderFunc which returns Hive compatible objects instead? We can save one conversion that way. e.g Avro objects -> Hive objects, instead of Avro objects -> Iceberg Generic objects -> Hive objects. The downside is we will have to build a Hive-compatible ReaderFunc for each file format but I don't expect it to be too hard.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448501109
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
I thought the convention was that the package name needed to match the subproject name and this isn't in the `hive` subproject but maybe that's not the case? Alternatively they could go in `org.apache.iceberg.mr.hive`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440055541
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
OK, sounds good, do you want to make this change as part of what you mention above with the Object Inspectors or later?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439509182
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
> Ultimately this all ends up being used by the old Map Reduce V1 API where serialization required the usage of Writables
I think the suggestion was to return `Record` from `deserialize`, not to use a non-writable. After `deserialize`, I don't think there is a requirement for anything to be a `Writable`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] guilload commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
guilload commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448509893
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
That made my life easier when adding new field but I get your point, I'll fix it in a follow-up PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438308102
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Test;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+
+public class TestIcebergSchemaToTypeInfo {
+
+ @Test
+ public void testGeneratePrimitiveTypeInfo() throws Exception {
+ Schema schema = new Schema(
+ required(1, "id", Types.IntegerType.get()),
+ optional(2, "data", Types.StringType.get()),
+ required(8, "feature1", Types.BooleanType.get()),
+ required(12, "lat", Types.FloatType.get()),
+ required(15, "x", Types.LongType.get()),
+ required(16, "date", Types.DateType.get()),
+ required(17, "double", Types.DoubleType.get()),
+ required(18, "binary", Types.BinaryType.get()));
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(8, types.size());
+ }
+
+ @Test
+ public void testGenerateMapTypeInfo() throws Exception {
+ TypeInfo expected = TypeInfoFactory.getMapTypeInfo(
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME));
+
+ Schema schema = new Schema(
+ optional(7, "properties", Types.MapType.ofOptional(18, 19,
+ Types.StringType.get(),
+ Types.StringType.get()
+ ), "string map of properties"));
+
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(1, types.size());
+ assertEquals(expected, types.get(0));
+ }
+
+ @Test
+ public void testGenerateListTypeInfo() throws Exception {
+ TypeInfo expected = TypeInfoFactory
+ .getListTypeInfo(TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME));
+ Schema schema = new Schema(
+ required(6, "doubles", Types.ListType.ofRequired(17,
+ Types.DoubleType.get()
+ )));
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(1, types.size());
+ assertEquals(expected, types.get(0));
+ }
+
+ @Test
+ public void testGenerateMapAndStructTypeInfo() throws Exception {
+ List<String> names1 = new ArrayList<>(Arrays.asList("address", "city", "state", "zip"));
+ List<TypeInfo> typeInfo1 = new ArrayList<>(Arrays.asList(
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME)
+ ));
+ TypeInfo mapKeyStructExpected = TypeInfoFactory.getStructTypeInfo(names1, typeInfo1);
+
+ List<String> names2 = new ArrayList<>(Arrays.asList("lat", "long"));
Review comment:
No need for `new ArrayList<>(...)` here. `Arrays.asList(...)` should return a list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438777741
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ }
+ case InputFormatConfig.HIVE_CATALOG:
+ //TODO Implement HiveCatalog
Review comment:
Hive adds a property to the conf called `name` which is whatever you've defined in the `CREATE TABLE name` part, so that property will include db and table name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448540762
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
Sounds good to me. You should be able to create and edit milestones.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438837722
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
IIRC when the InputFormat "runs" there is always a value in the associated HiveConf named "location" (`InputFormatConfig.TABLE_LOCATION` above) which points to the "base" location of the table being queried. So I think we should be able to just use that and require that the location is set correctly when the Hive table is created which is the normal way of doing things for external tables. Then we can just use that here and not worry about additional table properties or the other code paths which hopefully simplifies things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439190054
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
Review comment:
Maybe this can be a visitor? We recently built a visitor the other way round Hive -> Iceberg. https://github.com/linkedin/iceberg/blob/master/hive/src/main/java/org/apache/iceberg/hive/legacy/HiveTypeToIcebergType.java
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
Yea +1 for a visitor. We have TypeInfo to Iceberg Type visitor for inspiration https://github.com/linkedin/iceberg/blob/master/hive/src/main/java/org/apache/iceberg/hive/legacy/HiveTypeToIcebergType.java
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
Review comment:
throw Unsupported?
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
Review comment:
@cmathiesen @rdblue . Do u folks see value in having Hive classes in a separate hive module instead of MR? Similar to Iceberg Pig ? I think we need to provide a reader func implementation for Hive to be used with the MR module
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
+1. We should build this similar to reader Funcs. Please see examples of this in Spark module org.apache.iceberg.spark.data.SparkValueReaders [for Avro] and org.apache.iceberg.spark.data.SparkOrcValueReaders
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448811969
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
OK, my preference would be to leave this as it is for now and then do a review of all the packaging once we have the StorageHandler and InputFormat merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438895673
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
Review comment:
Hive uses `java.sql.Date` and `java.sql.Timestamp` so I've been working on doing some converting between types to stop Hive complaining :')
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438306862
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Test;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+
+public class TestIcebergSchemaToTypeInfo {
+
+ @Test
+ public void testGeneratePrimitiveTypeInfo() throws Exception {
+ Schema schema = new Schema(
+ required(1, "id", Types.IntegerType.get()),
+ optional(2, "data", Types.StringType.get()),
+ required(8, "feature1", Types.BooleanType.get()),
+ required(12, "lat", Types.FloatType.get()),
+ required(15, "x", Types.LongType.get()),
+ required(16, "date", Types.DateType.get()),
+ required(17, "double", Types.DoubleType.get()),
+ required(18, "binary", Types.BinaryType.get()));
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(8, types.size());
Review comment:
This should assert that the types are correct, not just that there is the correct number of types.
Also, please use context where possible to make it clear what the assertion expects, like "Converted TypeInfo should have the same number of columns". That makes it easier to understand when tests start to fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448476502
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergObjectInspector.java
##########
@@ -38,31 +38,37 @@
public class TestIcebergObjectInspector {
+ private int id = 0;
Review comment:
Why was this introduced? It seems like relying on the same execution order between the schema creation and the test methods is brittle.
I'd prefer to move back to fixed IDs since that's easier to test and more clear in assertions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-642173949
Thanks, @cmathiesen and @massdosage! Looks like a good start. I think we need to take a close look at how date/time types are handled because the Iceberg generics use Java 8 classes. I don't think that the object inspectors support those, and the inspector created for timestamp (converted to bigint `TypeInfo`) will definitely not work if it is called for a LocalDateTime or OffsetDateTime.
I'd also like to exercise the object inspectors in a test to make sure a record wrapped by IcebergWritable returns the correct types when deserialized and inspected. That would have caught the issue above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-640743706
This is required for #933 so that we can write proper integration tests for the IF with it using this SerDe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439463825
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
I don't think I understand what you want us to do here. We need it here for this to compile and get merged in separately from #933. We basically cut and paste the class out of the other branch to here. We'll happily resolve any conflicts etc. in #933 later if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448496908
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
Review comment:
can we put the hive classes in org.apache.iceberg.hive ? This is committed, but are you guys ok for this refactor?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438310960
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
Review comment:
Minor: you might consider changing the signature of `createCustomRecord` to `Schema, Object...` and wrap the object array as a list internally. Then these would be shorter: `createCustomRecord(schema, "Michael", 3000L)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438300385
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
Review comment:
It looks like this would be easier to implement using the type visitors, which already have the logic to traverse a schema. A good example is converting a [Type to Spark's DataType](https://github.com/apache/iceberg/blob/master/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java).
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
Review comment:
Doesn't Hive have timestamp types?
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, SystemTableUtil.getVirtualColumnName(serDeProperties),
Review comment:
I think `getVritualColumnName` should have a better method name. Here, it isn't clear what's happening because which virtual column is getting added is not obvious. If this were `snapshotIdColumnName` then I think it would be better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-652416692
@rdblue thanks for the comments in the review, we've implemented or replied to all of them. Could you please take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441522404
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME,
+ conf.get(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES)); //Default to HadoopTables
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES);
+ String tableLocation = properties.getProperty(InputFormatConfig.TABLE_LOCATION);
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ Preconditions.checkNotNull(tableLocation, "Table location is not set.");
+ Preconditions.checkNotNull(tableName, "Table name is not set.");
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
Review comment:
We've made a note of all of these suggestions so thank you, first of all, and we're working on a new implementation for system tables via Hive. As was [mentioned](https://github.com/apache/iceberg/pull/933#discussion_r439643514) in the other PR, we'll take out all the system table + virtual column stuff from these initial PR's and will open another PR for just system tables after merging this and #933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441650372
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -517,20 +411,20 @@ private Record withIdentityPartitionColumns(
}
private static Table findTable(Configuration conf) {
- String path = conf.get(TABLE_PATH);
+ String path = conf.get(InputFormatConfig.TABLE_PATH);
Preconditions.checkArgument(path != null, "Table path should not be null");
if (path.contains("/")) {
HadoopTables tables = new HadoopTables(conf);
return tables.load(path);
}
- String catalogFuncClass = conf.get(CATALOG);
+ String catalogFuncClass = conf.get(InputFormatConfig.CATALOG);
if (catalogFuncClass != null) {
Function<Configuration, Catalog> catalogFunc = (Function<Configuration, Catalog>)
- DynConstructors.builder(Function.class)
- .impl(catalogFuncClass)
- .build()
- .newInstance();
+ DynConstructors.builder(Function.class)
+ .impl(catalogFuncClass)
+ .build()
+ .newInstance();
Review comment:
Yes, definitely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438776518
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
We had initially implemented the IF using a method similar to what you're describing: we required a user to add an extra table property called `iceberg.warehouse.location` I think and used that to resolve the table with HadoopCatalog. The `LOCATION` property of your Hive table has to be unique for each table (which is why we always have a property for a full table path), so we couldn't put a path to a common warehouse there. We initially tried using the same location for multiple tables and relied on the table name/ID to resolve the table correctly in Iceberg but that caused major confusion in Hive when trying to join tables :')
So originally we used an extra property for the warehouse location, but we realised we could extract this location from the full table path and save the user an extra configuration property. We don't have any strong preference either way, so I'm happy to change this as you've suggested
Although thinking about this raises another point - I've just realised/tested that you could create an Iceberg table using HadoopCatalog but if you had the full path you can use HadoopTables to load the table again instead... And with Hive we can always expect to have the full path to the table because the location needs to be unique, so we could simplify here and only use HadoopTables for this IF? @massdosage
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439073269
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
Review comment:
It sounds like @guilload can help out with some of the object inspector questions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438313978
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergObjectInspectorGenerator.java
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+
+class IcebergObjectInspectorGenerator {
+
+ protected ObjectInspector createObjectInspector(Schema schema) throws Exception {
+ List<String> columnNames = setColumnNames(schema);
+ List<TypeInfo> columnTypes = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ List<ObjectInspector> columnOIs = new ArrayList<>(columnTypes.size());
+ for (int i = 0; i < columnTypes.size(); i++) {
+ columnOIs.add(createObjectInspectorWorker(columnTypes.get(i)));
+ }
+ return ObjectInspectorFactory.getStandardStructObjectInspector(columnNames, columnOIs, null);
+ }
+
+ protected ObjectInspector createObjectInspectorWorker(TypeInfo typeInfo) throws Exception {
+ ObjectInspector.Category typeCategory = typeInfo.getCategory();
+
+ switch (typeCategory) {
+ case PRIMITIVE:
+ PrimitiveTypeInfo pti = (PrimitiveTypeInfo) typeInfo;
+ return PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(pti);
Review comment:
Does Hive use `LocalDate` as its internal representation for `date`? That's what Iceberg generics are going to return. I think the object inspector needs to convert that correctly to the representation that Hive expects.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447350997
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergTimestampObjectInspector.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.sql.Timestamp;
+import java.time.LocalDateTime;
+import java.time.OffsetDateTime;
+import java.util.function.Function;
+import org.apache.hadoop.hive.serde2.io.TimestampWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+
+public final class IcebergTimestampObjectInspector extends IcebergPrimitiveObjectInspector
+ implements TimestampObjectInspector {
+
+ private static final IcebergTimestampObjectInspector INSTANCE_WITH_ZONE =
+ new IcebergTimestampObjectInspector(o -> ((OffsetDateTime) o).toLocalDateTime());
Review comment:
It seems like this would be a bit cleaner if the outer class was abstract and these were anonymous classes with an implementation for `LocalDateTime convert(Object o)` or something similar. Using `Function` is okay, but seems like it uses functions to avoid normal inheritance.
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergTimestampObjectInspector.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.sql.Timestamp;
+import java.time.LocalDateTime;
+import java.time.OffsetDateTime;
+import java.util.function.Function;
+import org.apache.hadoop.hive.serde2.io.TimestampWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+
+public final class IcebergTimestampObjectInspector extends IcebergPrimitiveObjectInspector
+ implements TimestampObjectInspector {
+
+ private static final IcebergTimestampObjectInspector INSTANCE_WITH_ZONE =
+ new IcebergTimestampObjectInspector(o -> ((OffsetDateTime) o).toLocalDateTime());
Review comment:
Minor: It seems like this would be a bit cleaner if the outer class was abstract and these were anonymous classes with an implementation for `LocalDateTime convert(Object o)` or something similar. Using `Function` is okay, but seems like it uses functions to avoid normal inheritance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438318069
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ }
+ case InputFormatConfig.HIVE_CATALOG:
+ //TODO Implement HiveCatalog
Review comment:
How are the db and table name passed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439073473
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
+ Map<String, String> expected = ImmutableMap.of("foo", "bar");
+ List<Map> data = new ArrayList<>();
+ data.add(expected);
+
+ Record record = TestHelpers.createCustomRecord(schema, data);
+ IcebergWritable writable = new IcebergWritable();
+ writable.setRecord(record);
+ writable.setSchema(schema);
+
+ IcebergSerDe serDe = new IcebergSerDe();
+ List<Object> deserialized = (List<Object>) serDe.deserialize(writable);
+ Map result = (Map) deserialized.get(0);
+
+ assertEquals(expected, result);
+ assertTrue(result.containsKey("foo"));
+ assertTrue(result.containsValue("bar"));
+ }
+
+ @Test
+ public void testDeserializeList() {
Review comment:
And if we take the advice from @guilload and just return the record, then we don't need to test at all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438311660
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
Review comment:
Does Hive support non-string map keys? It looks like all of the test cases are string maps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448477463
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestTableResolver.java
##########
@@ -111,4 +111,21 @@ public void resolveTableFromPropertiesDefault() throws IOException {
Assert.assertEquals(tableLocation.getAbsolutePath(), table.location());
}
+ @Test(expected = UnsupportedOperationException.class)
+ public void resolveTableFromConfigurationHiveCatalog() throws IOException {
+ Configuration conf = new Configuration();
+ conf.set(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HIVE_CATALOG);
+ conf.set(InputFormatConfig.TABLE_NAME, "table_a");
+
+ TableResolver.resolveTableFromConfiguration(conf);
+ }
+
+ @Test(expected = NullPointerException.class)
Review comment:
We usually prefer using `AssertHelpers.assertThrows` here, but this is minor since you don't need to check that other state has not been modified after the failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438312539
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
+ Map<String, String> expected = ImmutableMap.of("foo", "bar");
+ List<Map> data = new ArrayList<>();
+ data.add(expected);
+
+ Record record = TestHelpers.createCustomRecord(schema, data);
+ IcebergWritable writable = new IcebergWritable();
+ writable.setRecord(record);
+ writable.setSchema(schema);
+
+ IcebergSerDe serDe = new IcebergSerDe();
+ List<Object> deserialized = (List<Object>) serDe.deserialize(writable);
+ Map result = (Map) deserialized.get(0);
+
+ assertEquals(expected, result);
+ assertTrue(result.containsKey("foo"));
+ assertTrue(result.containsValue("bar"));
Review comment:
I think the `assertEquals` is sufficient since it checks deep equality.
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
+ Map<String, String> expected = ImmutableMap.of("foo", "bar");
+ List<Map> data = new ArrayList<>();
+ data.add(expected);
+
+ Record record = TestHelpers.createCustomRecord(schema, data);
+ IcebergWritable writable = new IcebergWritable();
+ writable.setRecord(record);
+ writable.setSchema(schema);
+
+ IcebergSerDe serDe = new IcebergSerDe();
+ List<Object> deserialized = (List<Object>) serDe.deserialize(writable);
+ Map result = (Map) deserialized.get(0);
Review comment:
Should this use the object inspectors?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447345753
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ }
+ case InputFormatConfig.HIVE_CATALOG:
+ //TODO Implement HiveCatalog
Review comment:
Should this throw an exception instead of returning null?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438287348
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Class to convert Iceberg types to Hive TypeInfo
+ */
+final class IcebergSchemaToTypeInfo {
+
+ private IcebergSchemaToTypeInfo() {
+
+ }
+
+ private static final ImmutableMap<Object, Object> primitiveTypeToTypeInfo = ImmutableMap.builder()
+ .put(Types.BooleanType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BOOLEAN_TYPE_NAME))
+ .put(Types.IntegerType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME))
+ .put(Types.LongType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.FloatType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.FLOAT_TYPE_NAME))
+ .put(Types.DoubleType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME))
+ .put(Types.BinaryType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BINARY_TYPE_NAME))
+ .put(Types.StringType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME))
+ .put(Types.DateType.get(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DATE_TYPE_NAME))
+ .put(Types.TimestampType.withoutZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .put(Types.TimestampType.withZone(), TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.BIGINT_TYPE_NAME))
+ .build();
+
+ public static List<TypeInfo> getColumnTypes(Schema schema) throws Exception {
+ List<Types.NestedField> fields = schema.columns();
+ List<TypeInfo> types = new ArrayList<>(fields.size());
+ for (Types.NestedField field : fields) {
+ types.add(generateTypeInfo(field.type()));
+ }
+ return types;
+ }
+
+ private static TypeInfo generateTypeInfo(Type type) throws Exception {
+ if (primitiveTypeToTypeInfo.containsKey(type)) {
+ return (TypeInfo) primitiveTypeToTypeInfo.get(type);
+ }
+ switch (type.typeId()) {
+ case UUID:
+ return TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME);
+ case FIXED:
+ return TypeInfoFactory.getPrimitiveTypeInfo("binary");
+ case TIME:
+ return TypeInfoFactory.getPrimitiveTypeInfo("long");
Review comment:
Why not use `serdeConstants` here and for the fixed case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441644752
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -265,9 +158,9 @@ private static void checkResiduals(CombinedScanTask task) {
Expression residual = fileScanTask.residual();
if (residual != null && !residual.equals(Expressions.alwaysTrue())) {
throw new UnsupportedOperationException(
- String.format(
- "Filter expression %s is not completely satisfied. Additional rows " +
- "can be returned not satisfied by the filter expression", residual));
+ String.format(
+ "Filter expression %s is not completely satisfied. Additional rows " +
+ "can be returned not satisfied by the filter expression", residual));
Review comment:
Yes, sorry!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-646812519
Hello @guilload, if we give you write access as an external collaborator to our fork of Iceberg at https://github.com/ExpediaGroup/iceberg would that make it easier to get your changes into this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438309982
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSchemaToTypeInfo.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Types;
+import org.junit.Test;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+
+public class TestIcebergSchemaToTypeInfo {
+
+ @Test
+ public void testGeneratePrimitiveTypeInfo() throws Exception {
+ Schema schema = new Schema(
+ required(1, "id", Types.IntegerType.get()),
+ optional(2, "data", Types.StringType.get()),
+ required(8, "feature1", Types.BooleanType.get()),
+ required(12, "lat", Types.FloatType.get()),
+ required(15, "x", Types.LongType.get()),
+ required(16, "date", Types.DateType.get()),
+ required(17, "double", Types.DoubleType.get()),
+ required(18, "binary", Types.BinaryType.get()));
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(8, types.size());
+ }
+
+ @Test
+ public void testGenerateMapTypeInfo() throws Exception {
+ TypeInfo expected = TypeInfoFactory.getMapTypeInfo(
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME));
+
+ Schema schema = new Schema(
+ optional(7, "properties", Types.MapType.ofOptional(18, 19,
+ Types.StringType.get(),
+ Types.StringType.get()
+ ), "string map of properties"));
+
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(1, types.size());
+ assertEquals(expected, types.get(0));
+ }
+
+ @Test
+ public void testGenerateListTypeInfo() throws Exception {
+ TypeInfo expected = TypeInfoFactory
+ .getListTypeInfo(TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.DOUBLE_TYPE_NAME));
+ Schema schema = new Schema(
+ required(6, "doubles", Types.ListType.ofRequired(17,
+ Types.DoubleType.get()
+ )));
+ List<TypeInfo> types = IcebergSchemaToTypeInfo.getColumnTypes(schema);
+
+ assertEquals(1, types.size());
+ assertEquals(expected, types.get(0));
+ }
+
+ @Test
+ public void testGenerateMapAndStructTypeInfo() throws Exception {
+ List<String> names1 = new ArrayList<>(Arrays.asList("address", "city", "state", "zip"));
+ List<TypeInfo> typeInfo1 = new ArrayList<>(Arrays.asList(
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME),
+ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.INT_TYPE_NAME)
+ ));
+ TypeInfo mapKeyStructExpected = TypeInfoFactory.getStructTypeInfo(names1, typeInfo1);
+
+ List<String> names2 = new ArrayList<>(Arrays.asList("lat", "long"));
+ List<TypeInfo> typeInfo2 = new ArrayList<>(Arrays.asList(
Review comment:
Minor: these names could be better to make it clear what's happening. These are the types for the value struct, so you could name the variable `valueTypes`, and the names `valueNames`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438659604
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
This is from #933, specifically at https://github.com/apache/iceberg/pull/933/files#diff-0aa47c0ca21b715226d83ae55f4b4671 so we just pulled that class in here "as is" as it was easier. If you'd prefer we can trim this down to just the config needed by the SerDe or we can (temporarily) put the code elsewhere in this PR and then refactor it back into this class when we come back to#933 after this is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r441211702
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -265,9 +158,9 @@ private static void checkResiduals(CombinedScanTask task) {
Expression residual = fileScanTask.residual();
if (residual != null && !residual.equals(Expressions.alwaysTrue())) {
throw new UnsupportedOperationException(
- String.format(
- "Filter expression %s is not completely satisfied. Additional rows " +
- "can be returned not satisfied by the filter expression", residual));
+ String.format(
+ "Filter expression %s is not completely satisfied. Additional rows " +
+ "can be returned not satisfied by the filter expression", residual));
Review comment:
I think the indentation was correct before. Can you revert this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439072297
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Properties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME, extractProperty(conf, InputFormatConfig.CATALOG_NAME));
+ if (conf.get(InputFormatConfig.CATALOG_NAME).equals(InputFormatConfig.HADOOP_CATALOG)) {
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE, conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ }
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME);
+ URI tableLocation = pathAsURI(properties.getProperty(InputFormatConfig.TABLE_LOCATION));
+ if (catalogName == null) {
+ throw new IllegalArgumentException("Catalog property: 'iceberg.catalog' not set in JobConf");
+ }
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ return tables.load(tableLocation.getPath());
+ case InputFormatConfig.HADOOP_CATALOG:
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ TableIdentifier id = TableIdentifier.parse(tableName);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
+ if (!Boolean.parseBoolean(properties.getProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ Boolean.TRUE.toString()))) {
+ String tablePath = id.toString().replaceAll("\\.", "/");
+ URI warehouseLocation = pathAsURI(tableLocation.getPath().replaceAll(tablePath, ""));
+ HadoopCatalog catalog = new HadoopCatalog(conf, warehouseLocation.getPath());
+ return catalog.loadTable(id);
+ } else {
+ return resolveMetadataTable(conf, tableLocation.getPath(), tableName);
+ }
+ } else {
+ URI warehouseLocation = pathAsURI(extractWarehousePath(tableLocation.getPath(), tableName));
Review comment:
> I've just realised/tested that you could create an Iceberg table using HadoopCatalog but if you had the full path you can use HadoopTables to load the table again instead...
Yes, this is why I said there is no benefit to using HadoopCatalog. If you already have the full path, you can open the table with HadoopTables and not worry about the warehouse path to pass to the catalog.
The only drawback is that this approach doesn't work for Hive tables. You'd need to know whether the table is a Hive table or a Hadoop table because they use different commit mechanisms.
For tables tracked by the iceberg-hive connector, you'd just need to instantiate the HiveCatalog and use the table name to load the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#issuecomment-652519017
I had a couple of minor comments, but we can fix those later. I'm going to merge this to unblock the next steps. Thanks @cmathiesen, @massdosage, @guilload, and everyone that reviewed!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r447364060
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/serde/objectinspector/TestIcebergBinaryObjectInspector.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.nio.ByteBuffer;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.hadoop.io.BytesWritable;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestIcebergBinaryObjectInspector {
+
+ @Test
+ public void testIcebergBinaryObjectInspector() {
Review comment:
It would be nice to have more cases in this test suite:
* When the buffer's limit is less than array().length
* When the buffer's arrayOffset is non-zero
* When the buffer's position is non-zero
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r439510074
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/IcebergSerDe.java
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.OffsetDateTime;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import javax.annotation.Nullable;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.serde2.AbstractSerDe;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeStats;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.io.Writable;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SnapshotsTable;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+
+public class IcebergSerDe extends AbstractSerDe {
+
+ private Schema schema;
+ private ObjectInspector inspector;
+ private List<Object> row;
+
+ @Override
+ public void initialize(@Nullable Configuration configuration, Properties serDeProperties) throws SerDeException {
+ Table table = null;
+ try {
+ table = TableResolver.resolveTableFromConfiguration(configuration, serDeProperties);
+ } catch (IOException e) {
+ throw new UncheckedIOException("Unable to resolve table from configuration: ", e);
+ }
+ this.schema = table.schema();
+ if (table instanceof SnapshotsTable) {
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(schema);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ } else {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE,
+ SystemTableUtil.snapshotIdVirtualColumnName(serDeProperties), Types.LongType.get()));
+ Schema withVirtualColumn = new Schema(columns);
+ try {
+ this.inspector = new IcebergObjectInspectorGenerator().createObjectInspector(withVirtualColumn);
+ } catch (Exception e) {
+ throw new SerDeException(e);
+ }
+ }
+ }
+
+ @Override
+ public Class<? extends Writable> getSerializedClass() {
+ return null;
+ }
+
+ @Override
+ public Writable serialize(Object o, ObjectInspector objectInspector) {
+ return null;
+ }
+
+ @Override
+ public SerDeStats getSerDeStats() {
+ return null;
+ }
+
+ @Override
+ public Object deserialize(Writable writable) {
Review comment:
> Instead of using ObjectInspectors, do you think it's worth building a ReaderFunc which returns Hive compatible objects instead?
This seems like a larger effort that is probably something for a follow-up instead. It seems much easier to me to add a small translation layer just for the types that need it and use the current Iceberg generics, rather than building a new object model for data formats.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440145422
##########
File path: mr/src/main/java/org/apache/iceberg/mr/InputFormatConfig.java
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr;
+
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SchemaParser;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.expressions.Expression;
+
+public class InputFormatConfig {
Review comment:
Ah, OK, now I see what you mean, yes, that makes sense. I've done that [now](https://github.com/apache/iceberg/pull/1103/commits/2350025f11bf3ce6df226d21a87d5c403ab50b53)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438688881
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
+
+ private SystemTableUtil() {}
+
+ protected static Schema schemaWithVirtualColumn(Schema schema, String columnName) {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, columnName, Types.LongType.get()));
+ return new Schema(columns);
+ }
+
+ protected static Record recordWithVirtualColumn(Record record, long snapshotId, Schema oldSchema,
+ String columnName) {
+ Schema newSchema = schemaWithVirtualColumn(oldSchema, columnName);
+ Record newRecord = GenericRecord.create(newSchema);
+ for (Types.NestedField field : oldSchema.columns()) {
+ newRecord.setField(field.name(), record.getField(field.name()));
+ }
+ newRecord.setField(columnName, snapshotId);
+ return newRecord;
+ }
+
+ protected static String getVirtualColumnName(Configuration conf) {
+ String virtualColumnName = conf.get(VIRTUAL_COLUMN_NAME);
+ if (virtualColumnName == null) {
+ return DEFAULT_SNAPSHOT_ID_COLUMN_NAME;
+ } else {
+ return virtualColumnName;
+ }
+ }
+
+ protected static String getVirtualColumnName(Properties properties) {
Review comment:
Yeah we agree, we discovered this when adding the `SerDe` - the IF uses `Configuration` but the `SerDe` only uses `Properties` and we wanted to use the methods across both classes and it seemed simpler to overload a method rather than create new `Properties` from the `Configuration` in the IF
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cmathiesen commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
cmathiesen commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438693642
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/SystemTableUtil.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.types.Types;
+
+public class SystemTableUtil {
+
+ static final String VIRTUAL_COLUMN_NAME = "iceberg.hive.snapshot.virtual.column.name";
+
+ private static final String DEFAULT_SNAPSHOT_ID_COLUMN_NAME = "snapshot__id";
+
+ private SystemTableUtil() {}
+
+ protected static Schema schemaWithVirtualColumn(Schema schema, String columnName) {
+ List<Types.NestedField> columns = new ArrayList<>(schema.columns());
+ columns.add(Types.NestedField.optional(Integer.MAX_VALUE, columnName, Types.LongType.get()));
+ return new Schema(columns);
+ }
+
+ protected static Record recordWithVirtualColumn(Record record, long snapshotId, Schema oldSchema,
+ String columnName) {
+ Schema newSchema = schemaWithVirtualColumn(oldSchema, columnName);
+ Record newRecord = GenericRecord.create(newSchema);
+ for (Types.NestedField field : oldSchema.columns()) {
+ newRecord.setField(field.name(), record.getField(field.name()));
+ }
+ newRecord.setField(columnName, snapshotId);
+ return newRecord;
+ }
+
+ protected static String getVirtualColumnName(Configuration conf) {
+ String virtualColumnName = conf.get(VIRTUAL_COLUMN_NAME);
+ if (virtualColumnName == null) {
+ return DEFAULT_SNAPSHOT_ID_COLUMN_NAME;
+ } else {
+ return virtualColumnName;
+ }
+ }
+
+ protected static String getVirtualColumnName(Properties properties) {
Review comment:
Although is exactly what we're doing in the `TableResolver` class... :')
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] teabot commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
teabot commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r440028833
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/TableResolver.java
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.IOException;
+import java.util.Properties;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.exceptions.NoSuchTableException;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.mr.InputFormatConfig;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+final class TableResolver {
+
+ private TableResolver() {
+ }
+
+ static Table resolveTableFromJob(JobConf conf) throws IOException {
+ Properties properties = new Properties();
+ properties.setProperty(InputFormatConfig.CATALOG_NAME,
+ conf.get(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES)); //Default to HadoopTables
+ properties.setProperty(InputFormatConfig.SNAPSHOT_TABLE,
+ conf.get(InputFormatConfig.SNAPSHOT_TABLE, "true"));
+ properties.setProperty(InputFormatConfig.TABLE_LOCATION, extractProperty(conf, InputFormatConfig.TABLE_LOCATION));
+ properties.setProperty(InputFormatConfig.TABLE_NAME, extractProperty(conf, InputFormatConfig.TABLE_NAME));
+ return resolveTableFromConfiguration(conf, properties);
+ }
+
+ static Table resolveTableFromConfiguration(Configuration conf, Properties properties) throws IOException {
+ String catalogName = properties.getProperty(InputFormatConfig.CATALOG_NAME, InputFormatConfig.HADOOP_TABLES);
+ String tableLocation = properties.getProperty(InputFormatConfig.TABLE_LOCATION);
+ String tableName = properties.getProperty(InputFormatConfig.TABLE_NAME);
+ Preconditions.checkNotNull(tableLocation, "Table location is not set.");
+ Preconditions.checkNotNull(tableName, "Table name is not set.");
+ switch (catalogName) {
+ case InputFormatConfig.HADOOP_TABLES:
+ HadoopTables tables = new HadoopTables(conf);
+ if (tableName.endsWith(InputFormatConfig.SNAPSHOT_TABLE_SUFFIX)) {
Review comment:
I definitely think we should adopt the simplification offered by points 2,3,4. My only concern with point 1, is that it can often be useful to guide users into a good convention by making it a default behaviour. That said, table naming strategies are almost certainly the responsibility of the data lake owner and not the implementation - so perhaps best to drop the reserved suffix also. Thanks for this suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] massdosage commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
massdosage commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r448359832
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapred/serde/objectinspector/IcebergObjectInspector.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred.serde.objectinspector;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+public final class IcebergObjectInspector extends TypeUtil.SchemaVisitor<ObjectInspector> {
+
+ public static ObjectInspector create(@Nullable Schema schema) {
+ if (schema == null) {
+ return IcebergRecordObjectInspector.empty();
+ }
+
+ return TypeUtil.visit(schema, new IcebergObjectInspector());
+ }
+
+ public static ObjectInspector create(Types.NestedField... fields) {
+ return create(new Schema(fields));
+ }
+
+ @Override
+ public ObjectInspector field(Types.NestedField field, ObjectInspector fieldObjectInspector) {
+ return fieldObjectInspector;
+ }
+
+ @Override
+ public ObjectInspector list(Types.ListType listTypeInfo, ObjectInspector listObjectInspector) {
+ return ObjectInspectorFactory.getStandardListObjectInspector(listObjectInspector);
+ }
+
+ @Override
+ public ObjectInspector map(Types.MapType mapType,
+ ObjectInspector keyObjectInspector, ObjectInspector valueObjectInspector) {
+ return ObjectInspectorFactory.getStandardMapObjectInspector(keyObjectInspector, valueObjectInspector);
+ }
+
+ @Override
+ public ObjectInspector primitive(Type.PrimitiveType primitiveType) {
+ final PrimitiveTypeInfo primitiveTypeInfo;
+
+ switch (primitiveType.typeId()) {
+ case BINARY:
+ return IcebergBinaryObjectInspector.get();
+ case BOOLEAN:
+ primitiveTypeInfo = TypeInfoFactory.booleanTypeInfo;
+ break;
+ case DATE:
+ return IcebergDateObjectInspector.get();
+ case DECIMAL:
+ Types.DecimalType type = (Types.DecimalType) primitiveType;
+ return IcebergDecimalObjectInspector.get(type.precision(), type.scale());
+ case DOUBLE:
+ primitiveTypeInfo = TypeInfoFactory.doubleTypeInfo;
+ break;
+ case FLOAT:
+ primitiveTypeInfo = TypeInfoFactory.floatTypeInfo;
+ break;
+ case INTEGER:
+ primitiveTypeInfo = TypeInfoFactory.intTypeInfo;
+ break;
+ case LONG:
+ primitiveTypeInfo = TypeInfoFactory.longTypeInfo;
+ break;
+ case STRING:
+ primitiveTypeInfo = TypeInfoFactory.stringTypeInfo;
+ break;
+ case TIMESTAMP:
+ boolean adjustToUTC = ((Types.TimestampType) primitiveType).shouldAdjustToUTC();
+ return IcebergTimestampObjectInspector.get(adjustToUTC);
+
+ case FIXED:
+ case TIME:
+ case UUID:
+ default:
+ throw new IllegalArgumentException(primitiveType.typeId() + " type is not supported");
Review comment:
First two done. As for `TIME` - Hive supports `DATE` and `TIMESTAMP`, I don't know enough about Iceberg's types to comment on how these differ from `TIME` but I'm guessing it doesn't?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1103: Add IcebergSerDe
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1103:
URL: https://github.com/apache/iceberg/pull/1103#discussion_r438313112
##########
File path: mr/src/test/java/org/apache/iceberg/mr/mapred/TestIcebergSerDe.java
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapred;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.hadoop.HadoopCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class TestIcebergSerDe {
+
+ private File tableLocation;
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ @Before
+ public void before() throws IOException {
+ tableLocation = temp.newFolder();
+ Schema schema = new Schema(optional(1, "name", Types.StringType.get()),
+ optional(2, "salary", Types.LongType.get()));
+ PartitionSpec spec = PartitionSpec.unpartitioned();
+
+ Configuration conf = new Configuration();
+ HadoopCatalog catalog = new HadoopCatalog(conf, tableLocation.getAbsolutePath());
+ TableIdentifier id = TableIdentifier.parse("source_db.table_a");
+ Table table = catalog.createTable(id, schema, spec);
+
+ List<Record> data = new ArrayList<>();
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Michael", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Andy", 3000L)));
+ data.add(TestHelpers.createCustomRecord(schema, Arrays.asList("Berta", 4000L)));
+
+ DataFile fileA = TestHelpers.writeFile(temp.newFile(), table, null, FileFormat.PARQUET, data);
+
+ table.newAppend().appendFile(fileA).commit();
+ }
+
+ @Test
+ public void testDeserializeMap() {
+ Schema schema = new Schema(required(1, "map_type", Types.MapType
+ .ofRequired(18, 19, Types.StringType.get(), Types.StringType.get())));
+ Map<String, String> expected = ImmutableMap.of("foo", "bar");
+ List<Map> data = new ArrayList<>();
+ data.add(expected);
+
+ Record record = TestHelpers.createCustomRecord(schema, data);
+ IcebergWritable writable = new IcebergWritable();
+ writable.setRecord(record);
+ writable.setSchema(schema);
+
+ IcebergSerDe serDe = new IcebergSerDe();
+ List<Object> deserialized = (List<Object>) serDe.deserialize(writable);
+ Map result = (Map) deserialized.get(0);
+
+ assertEquals(expected, result);
+ assertTrue(result.containsKey("foo"));
+ assertTrue(result.containsValue("bar"));
+ }
+
+ @Test
+ public void testDeserializeList() {
Review comment:
I'm not sure so many test cases are necessary since this is really just testing that a record is converted to a collection with the same field order. That's all `deserialize` is doing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org