You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/02/17 21:14:41 UTC

[GitHub] [incubator-iceberg] sundargates opened a new pull request #808: Implementing support for modifying namemapping programmatically

sundargates opened a new pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808
 
 
   Currently, Iceberg doesn't support updating name mapping programmatically. This becomes especially hard when columns are added/removed as updating the name mapping involves messing with the field IDs. 
   
   This PR addresses the problem by
   a. Defining an API for updating the name mapping
   b. Providing an implementation for the aforementioned API

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on issue #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#issuecomment-610077379
 
 
   This hasn't been updated in 6 weeks since the review. I'm going to close it. Feel free to re-open when this is updated. Thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#discussion_r382754954
 
 

 ##########
 File path: core/src/main/java/org/apache/iceberg/NameMappingUpdate.java
 ##########
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.iceberg.mapping.MappedField;
+import org.apache.iceberg.mapping.MappedFields;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+/**
+ * NameMapping evolution API implementation.
+ */
+class NameMappingUpdate implements UpdateNameMapping {
+  private final Schema schema;
+  private final NameMapping nameMapping;
+  private final UpdateProperties updateProperties;
+  private final Map<Integer, Iterable<String>> adds = new ConcurrentHashMap<>();
+
+  NameMappingUpdate(TableOperations ops) {
+    TableMetadata tableMetadata = ops.current();
+    this.schema = tableMetadata.schema();
+    this.nameMapping = NameMappingParser.fromJson(tableMetadata.properties().get(TableProperties.DEFAULT_NAME_MAPPING));
+    this.updateProperties = new PropertiesUpdate(ops);
+  }
+
+  /**
+   * For testing only.
+   */
+  NameMappingUpdate(
+      Schema schema, NameMapping nameMapping, UpdateProperties updateProperties) {
+    this.schema = schema;
+    this.nameMapping = nameMapping;
+    this.updateProperties = updateProperties;
+  }
+
+  private void addAliases(int fieldId, Iterable<String> aliases) {
 
 Review comment:
   I also like to add `String...` variants of API methods that accept an `Iterable`, to save callers from coercing from array or creating a collection.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#discussion_r382759977
 
 

 ##########
 File path: core/src/main/java/org/apache/iceberg/NameMappingUpdate.java
 ##########
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.iceberg.mapping.MappedField;
+import org.apache.iceberg.mapping.MappedFields;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+/**
+ * NameMapping evolution API implementation.
+ */
+class NameMappingUpdate implements UpdateNameMapping {
+  private final Schema schema;
+  private final NameMapping nameMapping;
+  private final UpdateProperties updateProperties;
+  private final Map<Integer, Iterable<String>> adds = new ConcurrentHashMap<>();
+
+  NameMappingUpdate(TableOperations ops) {
+    TableMetadata tableMetadata = ops.current();
+    this.schema = tableMetadata.schema();
+    this.nameMapping = NameMappingParser.fromJson(tableMetadata.properties().get(TableProperties.DEFAULT_NAME_MAPPING));
+    this.updateProperties = new PropertiesUpdate(ops);
+  }
+
+  /**
+   * For testing only.
+   */
+  NameMappingUpdate(
+      Schema schema, NameMapping nameMapping, UpdateProperties updateProperties) {
+    this.schema = schema;
+    this.nameMapping = nameMapping;
+    this.updateProperties = updateProperties;
+  }
+
+  private void addAliases(int fieldId, Iterable<String> aliases) {
+    adds.compute(fieldId, (dontCare, oldValue) -> {
+      if (oldValue == null) {
+        return aliases;
+      } else {
+        return ImmutableList.<String>builder().addAll(aliases).build();
+      }
+    });
+  }
+
+  @Override
+  public UpdateNameMapping addAliases(String name, Iterable<String> aliases) {
 
 Review comment:
   There are some missing operations that we need for schema mapping. A mapping may be updated when a producer schema changes and that may not be reflected in the table schema.
   
   For example, if the schema of the incoming Kafka topic adds "z", then we should add that field to the mapping, indicating that there is a field available that hasn't yet been mapped. Then when "z" is added to the table schema, we can update the existing mapping. That operation requires adding a new mapping without an ID.
   
   For that use case, I think we should add `addField(Iterable<String> parent, String name)` and  `addField(String name)` (top-level). Then additional aliases could be added using these methods. And we could also add `addId(Iterable<String> name, int id)` to set the ID for an existing mapping.
   
   For the `addField` methods, if the field exists in the schema, we should automatically set its id when adding the mapping.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue closed pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue closed pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#discussion_r382760239
 
 

 ##########
 File path: core/src/main/java/org/apache/iceberg/NameMappingUpdate.java
 ##########
 @@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.iceberg.mapping.MappedField;
+import org.apache.iceberg.mapping.MappedFields;
+import org.apache.iceberg.mapping.NameMapping;
+import org.apache.iceberg.mapping.NameMappingParser;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.TypeUtil;
+import org.apache.iceberg.types.Types;
+
+/**
+ * NameMapping evolution API implementation.
+ */
+class NameMappingUpdate implements UpdateNameMapping {
+  private final Schema schema;
+  private final NameMapping nameMapping;
+  private final UpdateProperties updateProperties;
+  private final Map<Integer, Iterable<String>> adds = new ConcurrentHashMap<>();
+
+  NameMappingUpdate(TableOperations ops) {
+    TableMetadata tableMetadata = ops.current();
+    this.schema = tableMetadata.schema();
+    this.nameMapping = NameMappingParser.fromJson(tableMetadata.properties().get(TableProperties.DEFAULT_NAME_MAPPING));
+    this.updateProperties = new PropertiesUpdate(ops);
+  }
+
+  /**
+   * For testing only.
+   */
+  NameMappingUpdate(
+      Schema schema, NameMapping nameMapping, UpdateProperties updateProperties) {
+    this.schema = schema;
+    this.nameMapping = nameMapping;
+    this.updateProperties = updateProperties;
+  }
+
+  private void addAliases(int fieldId, Iterable<String> aliases) {
+    adds.compute(fieldId, (dontCare, oldValue) -> {
+      if (oldValue == null) {
+        return aliases;
+      } else {
+        return ImmutableList.<String>builder().addAll(aliases).build();
+      }
+    });
+  }
+
+  @Override
+  public UpdateNameMapping addAliases(String name, Iterable<String> aliases) {
+    Preconditions.checkArgument(schema.findField(name) != null, "Cannot find column: %s", name);
+    int fieldId = schema.findField(name).fieldId();
 
 Review comment:
   I don't think we should require a schema lookup here. When adding a field to the mapping, we can add the ID from the schema, but we shouldn't require a field in the schema to update the mapping. There's no need to tie the two together.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#discussion_r381026780
 
 

 ##########
 File path: api/src/main/java/org/apache/iceberg/Table.java
 ##########
 @@ -115,6 +115,13 @@
    */
   UpdateSchema updateSchema();
 
+  /**
+   * Create a new {@link UpdateNameMapping} to update name mapping of this table and commit the change.
+   *
+   * @return a new {@link UpdateNameMapping}
+   */
+  UpdateNameMapping updateNameMapping();
 
 Review comment:
   While I think it makes sense to update the mapping programmatically, I don't see much value in exposing it as part of the table API. We want to keep the Table API small, so if we can handle this by using a separate API that consumes and produces JSON, then that is preferred. Also, we may have more than one mapping in the future, so keeping this decoupled also helps out for those use cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically

Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #808: Implementing support for modifying namemapping programmatically
URL: https://github.com/apache/incubator-iceberg/pull/808#discussion_r381026780
 
 

 ##########
 File path: api/src/main/java/org/apache/iceberg/Table.java
 ##########
 @@ -115,6 +115,13 @@
    */
   UpdateSchema updateSchema();
 
+  /**
+   * Create a new {@link UpdateNameMapping} to update name mapping of this table and commit the change.
+   *
+   * @return a new {@link UpdateNameMapping}
+   */
+  UpdateNameMapping updateNameMapping();
 
 Review comment:
   While I think it makes sense to update the mapping programmatically, I don't see much value in exposing it as part of the table API. We want to keep the Table API small, so if we can handle this by using a separate API that consumes and produces JSON, then that is preferred. Also, we may have more than one mapping in the future, so keeping this decoupled also helps out for those use cases.
   
   For now, let's build a utility class to perform these updates. We can always add this to the public API later if we choose to.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org