You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/03/24 03:54:17 UTC
[GitHub] [iceberg] openinx commented on a change in pull request #2354: Core: add row identifier to format v2
openinx commented on a change in pull request #2354:
URL: https://github.com/apache/iceberg/pull/2354#discussion_r600134551
##########
File path: api/src/main/java/org/apache/iceberg/RowIdentifyField.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.types.Types;
+
+/**
+ * A field in {@link RowIdentifier}
+ * <p>
+ * The field must be:
+ * 1. a required column in the table schema
+ * 2. a primitive type column
+ */
+public class RowIdentifyField implements Serializable {
+
+ private final int sourceId;
+
+ RowIdentifyField(int sourceId) {
+ this.sourceId = sourceId;
+ }
+
+ RowIdentifyField(Types.NestedField column) {
+ ValidationException.check(column.isRequired(),
+ "Cannot add column %s to row identifier because it is not a required column", column);
+ ValidationException.check(column.type().isPrimitiveType(),
+ "Cannot add column %s to row identifier because it is not of a primitive data type", column);
Review comment:
Nit: should we remove the `of` in the error message ?
##########
File path: api/src/main/java/org/apache/iceberg/RowIdentifyField.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.types.Types;
+
+/**
+ * A field in {@link RowIdentifier}
+ * <p>
+ * The field must be:
+ * 1. a required column in the table schema
+ * 2. a primitive type column
+ */
+public class RowIdentifyField implements Serializable {
+
+ private final int sourceId;
+
+ RowIdentifyField(int sourceId) {
+ this.sourceId = sourceId;
+ }
+
+ RowIdentifyField(Types.NestedField column) {
+ ValidationException.check(column.isRequired(),
+ "Cannot add column %s to row identifier because it is not a required column", column);
Review comment:
Nit: In this `RowIdentifyField` constructor, we will throw the exception with message saying `Cannot add column ...` , that seems unreasonable because we are mixed the column adding and instance constructing together. How about moving those validation check out of this constructor ?
##########
File path: api/src/main/java/org/apache/iceberg/RowIdentifier.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Row identifier of a table.
+ * <p>
+ * Row identifier is a definition of table row uniqueness,
+ * similar to the concept of primary key in a relational database system.
+ * A row should be unique in a table based on the values of each {@link RowIdentifyField}.
+ * Iceberg itself does not enforce row uniqueness based on this identifier.
+ * It is leveraged by operations such as streaming upsert.
+ */
+public class RowIdentifier implements Serializable {
Review comment:
Yeah, the `Row identifier` is the correct semantic to express. On minor thing for me is : the name seems a bit long as a part of table spec, how about using `RowKey` as the name ?
##########
File path: api/src/main/java/org/apache/iceberg/RowIdentifier.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Row identifier of a table.
+ * <p>
+ * Row identifier is a definition of table row uniqueness,
+ * similar to the concept of primary key in a relational database system.
+ * A row should be unique in a table based on the values of each {@link RowIdentifyField}.
+ * Iceberg itself does not enforce row uniqueness based on this identifier.
+ * It is leveraged by operations such as streaming upsert.
+ */
+public class RowIdentifier implements Serializable {
+
+ private static final RowIdentifier NOT_IDENTIFIED = new RowIdentifier(null, 0, ImmutableList.of());
+
+ private final Schema schema;
+ private final int rowIdVersion;
Review comment:
If we plan to change the name as `RowKey`, then we could use the `rowKeyId` here.
##########
File path: api/src/main/java/org/apache/iceberg/RowIdentifyField.java
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.types.Types;
+
+/**
+ * A field in {@link RowIdentifier}
+ * <p>
+ * The field must be:
+ * 1. a required column in the table schema
+ * 2. a primitive type column
+ */
+public class RowIdentifyField implements Serializable {
Review comment:
I agreed that it's extensible to introduce a separate `RowIdentifyField` class here. For the `transform` function, currently I don't see there's any requirement that we will need it for this row identifier fields, but I'm OK to make it extensible for future usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org