You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/15 09:24:39 UTC
[GitHub] [hudi] xiarixiaoyao opened a new pull request #3668: [HUDI-2429][WIP] Full schema evolution
xiarixiaoyao opened a new pull request #3668:
URL: https://github.com/apache/hudi/pull/3668
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
introduce a id-schema as hoodie internalSchema to support full schema evolution:
Added columns never read existing values from another column.
Dropping a column or field does not change the values in any other column.
Updating a column or field does not change values in any other column.
Changing the order of columns or fields in a struct does not change the values associated with a column or field name
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715484436
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+/**
+ * helper class.
+ * use static cache to cache historySchema info to prevent fetching from hdfs every time.
+ * all task belong to the same executor/container will share this cache.
+ *
+ */
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
Review comment:
we need a statics cache which will be reused by JVM。 Fetch historySchema maybe time consuming， we need do cache for it。 we use tablePath+schema_version_Id as key to cache historySchema，which is safe for reusing。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709815227
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
+ * @param newName new name for current col. no need to specify fullName.
+ */
+ public void renameColumn(String colName, String newName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.renameColumn(colName, newName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col type for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param nullable .
+ */
+ public void updateColumnNullability(String colName, boolean nullable) {
Review comment:
no this method used to alter table column nullable attribute。 sorry， the annotated is wrong. will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64c7ecf585993377b6a0d9b18e14003068471e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919955374
@yanghua yes, We reached a preliminary consensus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709101008
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
Review comment:
PostitionType can be enum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2d8b3f67e3556ccb84309b381ea513731537f8b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400)
* b5e0b228840b7c0575b6eaaf888e0f5bb6024b91 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-951362976
This feature is amazing, long-awaited functionality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772)
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaf33e3a28680ab5febca7df70937ce543619a94 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717251070
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
ok， will do it. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaf33e3a28680ab5febca7df70937ce543619a94 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632)
* 8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772)
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r724179432
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
I think for implicit schema changes we would want to control whether to have current behavior or introduce the one. Same is true for the reader side.it is better to simply have a gatekeeper config to control the entire feature. when the feature matures, we can turn it on by default and deprecate later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89dab78876b2512aa4967ced70da27f6fdb46b14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89dab78876b2512aa4967ced70da27f6fdb46b14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619)
* aaf33e3a28680ab5febca7df70937ce543619a94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-920594203
> cc @codope @bvaradar lets use this as a basis and evolve our design, move forward.
>
> @yanghua we are close I think. Few open items, that can be resolved soon hopefully.
> You can follow along the discussion in the comments here
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution
>
> @codope also has a bunch of these working on presto and trino already. Flink would be a good thing to tackle as well. Does Flink use the Hive record readers? cc @danny0405 as well.
Flink does not use the Hive record readers now ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-926015942
Thanks @xiarixiaoyao , Yes, It makes sense to add all the spark adapations related changes to this PR .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709115319
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
Review comment:
ok, will change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717230330
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
@xiarixiaoyao : The synchronous metadata table support is still in the works and is getting ready. Its better to decouple both the projects and at the same time avoid adding historical schema to commit files. As an interim measure, lets write as separate files under .hoodie/schema and then we can do a follow up PR whenever synchronous metadata table is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717266758
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717298481
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordScanner.java
##########
@@ -45,7 +45,7 @@ private HoodieMetadataMergedLogRecordScanner(FileSystem fs, String basePath, Lis
String spillableMapBasePath, Set<String> mergeKeyFilter,
ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled) {
super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, false, false, bufferSize,
- spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false);
+ spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false, null);
Review comment:
spark mor/cow has not use HoodieUnmergedLogRecordScanner only hive/presto use it. this pr is only about spark adapation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717296982
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
done,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715455246
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -87,4 +88,9 @@ trait SparkAdapter extends Serializable {
* Create Like expression.
*/
def createLike(left: Expression, right: Expression): Expression
+
+ /**
+ * Create parquet file read for hudi.
+ */
+ def createHoodieParquetFileFormat(): ParquetFileFormat
Review comment:
ok, cow incremental/snapshort view and mor incremetal/optimize view will added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574)
* cdcf299c9f4e743cd74d9266603a30bf3301c3b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f0a8e1025051d6077e168b4ca15f4ee8edccc1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422)
* 42ca73a97f00023736feeda07e96d03e7a621c9b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r724179432
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
I think for implicit schema changes we would want to control whether to have current behavior or introduce the one. Same is true for the reader side.it is better to simply have a gatekeeper config to control the entire feature. when the feature matures, we can turn it on by default and deprecate later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a0b7d831b32d1704c1837a2df77bfe4d36724cc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-922247449
@leesf @codope @bvaradar update the code , add docs to all public method, and resolved most comments. pls help me review those code again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710887355
##########
File path: hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark2HoodieParquetFileFormat.scala
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.net.URI
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.mapreduce.lib.input.FileSplit
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.hadoop.mapreduce.{JobID, TaskAttemptID, TaskID, TaskType}
+import org.apache.hudi.client.utils.SparkSchemaUtils
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.util.TableInternalSchemaUtils
+import org.apache.hudi.internal.schema.InternalSchema
+import org.apache.hudi.internal.schema.utils.{InternalSchemaParser, InternalSchemaUtils}
+import org.apache.parquet.filter2.compat.FilterCompat
+import org.apache.parquet.filter2.predicate.FilterApi
+import org.apache.parquet.format.converter.ParquetMetadataConverter.SKIP_ROW_GROUPS
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetRecordReader}
+import org.apache.spark.TaskContext
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{JoinedRow, UnsafeRow}
+import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.execution.datasources.{PartitionedFile, RecordReaderIterator}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types.{AtomicType, StructType}
+import org.apache.spark.util.SerializableConfiguration
+
+class Spark2HoodieParquetFileFormat extends ParquetFileFormat {
+ // reference ParquetFileFormat from spark project
+ override def buildReaderWithPartitionValues(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ options: Map[String, String],
+ hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ if (hadoopConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA).isEmpty) {
+ // fallback to origin parquet File read
+ super.buildReaderWithPartitionValues(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf)
+ } else {
+ hadoopConf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ hadoopConf.set(
+ ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ ParquetWriteSupport.SPARK_ROW_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ SQLConf.SESSION_LOCAL_TIMEZONE.key,
+ sparkSession.sessionState.conf.sessionLocalTimeZone)
+ hadoopConf.setBoolean(
+ SQLConf.NESTED_SCHEMA_PRUNING_ENABLED.key,
+ sparkSession.sessionState.conf.nestedSchemaPruningEnabled)
+ hadoopConf.setBoolean(
+ SQLConf.CASE_SENSITIVE.key,
+ sparkSession.sessionState.conf.caseSensitiveAnalysis)
+
+ ParquetWriteSupport.setSchema(requiredSchema, hadoopConf)
+
+ // Sets flags for `ParquetToSparkSchemaConverter`
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_BINARY_AS_STRING.key,
+ sparkSession.sessionState.conf.isParquetBinaryAsString)
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP.key,
+ sparkSession.sessionState.conf.isParquetINT96AsTimestamp)
+
+ val broadcastedHadoopConf =
+ sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))
+
+ // TODO: if you move this into the closure it reverts to the default values.
+ // If true, enable using the custom RecordReader for parquet. This only works for
+ // a subset of the types (no complex types).
+ val resultSchema = StructType(partitionSchema.fields ++ requiredSchema.fields)
+ val sqlConf = sparkSession.sessionState.conf
+ val enableOffHeapColumnVector = sqlConf.offHeapColumnVectorEnabled
+ val enableVectorizedReader: Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ resultSchema.forall(_.dataType.isInstanceOf[AtomicType])
+ val enableRecordFilter: Boolean = sqlConf.parquetRecordFilterEnabled
+ val timestampConversion: Boolean = sqlConf.isParquetINT96TimestampConversion
+ val capacity = sqlConf.parquetVectorizedReaderBatchSize
+ val enableParquetFilterPushDown: Boolean = sqlConf.parquetFilterPushDown
+ // Whole stage codegen (PhysicalRDD) is able to deal with batches directly
+ val returningBatch = supportBatch(sparkSession, resultSchema)
+ val pushDownDate = sqlConf.parquetFilterPushDownDate
+ val pushDownTimestamp = sqlConf.parquetFilterPushDownTimestamp
+ val pushDownDecimal = sqlConf.parquetFilterPushDownDecimal
+ val pushDownStringStartWith = sqlConf.parquetFilterPushDownStringStartWith
+ val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
+ val isCaseSensitive = sqlConf.caseSensitiveAnalysis
+
+ (file: PartitionedFile) => {
+ assert(file.partitionValues.numFields == partitionSchema.size)
+
+ val fileSplit =
+ new FileSplit(new Path(new URI(file.filePath)), file.start, file.length, Array.empty)
+ val filePath = fileSplit.getPath
+
+ val split =
+ new org.apache.parquet.hadoop.ParquetInputSplit(
+ filePath,
+ fileSplit.getStart,
+ fileSplit.getStart + fileSplit.getLength,
+ fileSplit.getLength,
+ fileSplit.getLocations,
+ null)
+
+ val sharedConf = broadcastedHadoopConf.value.value
+
+ // do deal with internalSchema
+ val internalSchemaString = sharedConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA)
+ val internalSchemaChangeEnabled = if (internalSchemaString.isEmpty) false else true
+ // querySchema must be a pruned schema.
+ val querySchema = InternalSchemaParser.fromJson(internalSchemaString)
Review comment:
internalSchemaChangeEnabled will deal with this corner case， rebuildFilterFromParquet will also deal with this corner case， donot worried
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a0b7d831b32d1704c1837a2df77bfe4d36724cc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263)
* 5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba407cc5af64d50eaa18f5df4adf1307853c264d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709815503
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717191016
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
Review comment:
if we donot modify the table schema explicitly. LATESTSCHEMA is only produced when we do alter SQL or alter api.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fe30385aaabcc0b7f1474228d397ea67e174804 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460)
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-921467046
@codope thanks for your review. Let me answer some your questions first， I’m a bit busy today, I need to modify something for the pr of zorder
**Let's add docs for all the public classes and APIs.**
ok，i will added。
**Does the merge schema action handle evolution of non-leaf fields in nested fields? For example, if a.b.c is renamed to a.d.c.**
this is a strange demand， if you want to change a.b.c to a.d.c , why donot use spark.sql(s"alter table ${tableName} rename column a.b to d"). we support handle all the non-leaf fields.
**IIUC, the patch has not yet handled old or existing schema compatibility as mentioned in the RFC right?**
already deal with this situation. pls see the TestSpark3DDL, we first create a table and insert some data to it. now no id-schema is produced. then we do schema change, and the id-schema is produced. the test result confirmed this。
**Since schema history is being changed not only at the write time but also at the read time, so we need to think of both writer and reader concurrency.**
A little doubtful， hudi is snapshot isolated， maybe only need to deal with concurrency between write and write， if i am wrong pls fix me ， thanks
Now there is the most important question， I now tend to use metatable to store historical schema。 as we know metatable use hfile to store data， the hfile has a very good point query performance， what do you suggest？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42ca73a97f00023736feeda07e96d03e7a621c9b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459)
* 0fe30385aaabcc0b7f1474228d397ea67e174804 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709818122
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
+ HISTORICAL_SCHEMA_CACHE = Caffeine.newBuilder().maximumSize(1000).weakValues().build();
+
+ public static InternalSchema searchSchemaAndCache(long versionID, String tablePath, Configuration hadoopConf) {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715457071
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
ok, but now we need to reach agreement on this issue。 vinothchandar suggest we save those information into metatable。
@vinothchandar What is your opinion on this。
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1100,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717282617
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChanges.java
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.SchemaChangeUtils;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class TableChanges {
Review comment:
remove helper methods to a helper class. now TableChanges is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709165632
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
Review comment:
should be a `List<String> colNames`? or rename to `deleteColumn`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 101b85af0521d578a9a42eb50a8b523bc65d0abf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-961588408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-961588408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919861244
kindly ping @bvaradar @leesf could you help me review those code . thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-920619025
@danny0405 Hope you can help us complete the full schema evolution of flink， thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919980674
cc @codope @bvaradar lets use this as a basis and evolve our design, move forward.
@yanghua we are close I think. Few open items, that can be resolved soon hopefully.
You can follow along the discussion in the comments here
https://cwiki.apache.org/confluence/display/HUDI/RFC+-+33++Hudi+supports+more+comprehensive+Schema+Evolution
@codope also has a bunch of these working on presto and trino already. Flink would be a good thing to tackle as well. Does Flink use the Hive record readers? cc @danny0405 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709168436
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
+ * @param newName new name for current col. no need to specify fullName.
+ */
+ public void renameColumn(String colName, String newName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.renameColumn(colName, newName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col type for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param nullable .
+ */
+ public void updateColumnNullability(String colName, boolean nullable) {
Review comment:
this method is used to alter table column type such as from int to long or any other methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709101008
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
Review comment:
PositionType can be enum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64c7ecf585993377b6a0d9b18e14003068471e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709189528
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
Review comment:
docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r711563248
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
+ Integer nameId = querySchema.findIdByName(name);
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r724824932
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
agree, will deal it. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f0a8e1025051d6077e168b4ca15f4ee8edccc1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422)
* 42ca73a97f00023736feeda07e96d03e7a621c9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737)
* d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715459373
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkSchemaUtils.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.ArrayType$;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.BooleanType$;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.CharType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DecimalType$;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.DoubleType$;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.FloatType$;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.MapType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.sql.types.TimestampType$;
+import org.apache.spark.sql.types.UserDefinedType;
+import org.apache.spark.sql.types.VarcharType;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class SparkSchemaUtils {
+ private SparkSchemaUtils() {
+
+ }
+
+ public static final String HOODIE_QUERY_SCHEMA = "hadoop.hoodie.querySchema";
Review comment:
we need pass internalSchema to parquetFileReader（which run in executor not driver）， so we set those parameter to hadoopConf to pass those parameter to parquetFileReader。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709114920
##########
File path: hudi-common/pom.xml
##########
@@ -172,6 +172,12 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
Review comment:
used to cache the history schema. caffine is better than guava cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba407cc5af64d50eaa18f5df4adf1307853c264d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251)
* 1a0b7d831b32d1704c1837a2df77bfe4d36724cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b5e0b228840b7c0575b6eaaf888e0f5bb6024b91 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466)
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717275081
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
Review comment:
already done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717262692
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieUnMergedLogRecordScanner.java
##########
@@ -38,7 +38,7 @@
private HoodieUnMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
String latestInstantTime, boolean readBlocksLazily, boolean reverseReader, int bufferSize,
LogRecordScannerCallback callback, Option<InstantRange> instantRange) {
- super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false);
+ super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false, null);
Review comment:
no, spark mor/cow has not use HoodieUnmergedLogRecordScanner only hive/presto use it. this pr is only about spark adapation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717283589
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaUtil.java
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import static org.apache.avro.Schema.Type.UNION;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.action.TableChanges;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+public class AvroSchemaUtil {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717230330
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
@xiarixiaoyao : The synchronous metadata table support is still in the works and is getting ready. Its better to decouple both the projects and at the same time avoid adding historical schema to commit files. As an interim measure, lets write as separate files under .hoodie/schema and then we can do a follow up PR whenever synchronous metadata table is done.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
@xiarixiaoyao : The synchronous metadata table support is still in the works and is getting ready. Its better to decouple both the projects and at the same time avoid adding historical schema to commit files. As an interim measure, lets write as separate files under .hoodie/schema and then we can do a follow up PR whenever synchronous metadata table is done. cc @vinothchandar @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-930677502
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fe30385aaabcc0b7f1474228d397ea67e174804 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717299317
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala
##########
@@ -52,26 +54,33 @@ case class HoodieMergeOnReadTableState(tableStructSchema: StructType,
requiredAvroSchema: String,
hoodieRealtimeFileSplits: List[HoodieMergeOnReadFileSplit],
preCombineField: Option[String],
- recordKeyFieldOpt: Option[String])
+ recordKeyFieldOpt: Option[String],
+ internalSchema: Option[InternalSchema] = None,
+ requiredInternalSchema: Option[InternalSchema] = None)
class MergeOnReadSnapshotRelation(val sqlContext: SQLContext,
val optParams: Map[String, String],
val userSchema: StructType,
val globPaths: Option[Seq[Path]],
val metaClient: HoodieTableMetaClient)
- extends BaseRelation with PrunedFilteredScan with Logging {
+ extends BaseRelation with PrunedFilteredScan with Logging with SparkAdapterSupport {
private val conf = sqlContext.sparkContext.hadoopConfiguration
private val jobConf = new JobConf(conf)
// use schema from latest metadata, if not present, read schema from the data file
private val schemaUtil = new TableSchemaResolver(metaClient)
- private lazy val tableAvroSchema = {
+ private lazy val (tableAvroSchema, internalSchema) = {
try {
- schemaUtil.getTableAvroSchema
+ val internalSchemaOpt = schemaUtil.getTableInternalSchemaFromCommitMetadata
+ if (internalSchemaOpt.isPresent) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64c7ecf585993377b6a0d9b18e14003068471e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235)
* 101b85af0521d578a9a42eb50a8b523bc65d0abf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710873627
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -87,4 +88,9 @@ trait SparkAdapter extends Serializable {
* Create Like expression.
*/
def createLike(left: Expression, right: Expression): Expression
+
+ /**
+ * Create parquet file read for hudi.
+ */
+ def createHoodieParquetFileFormat(): ParquetFileFormat
Review comment:
yes，we need do that。but i think this pr is more like the prototype of id-schema, I hope we can reach a basic agreement on this or。 full spark adaption will be another pr 。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285)
* b2d8b3f67e3556ccb84309b381ea513731537f8b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709165632
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
Review comment:
should be a `List<String> colNames`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717191016
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
Review comment:
if we donot modify the table schema explicitly. LATESTSCHEMA is only produced when we do alter SQL or alter api.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
> We need to introduce a config to enable/disable schema evolution. Only if the config is enables, we should let any side-effects of schema evolution to take effect.
may no need introduce a config. only when user execute alter SQL /alter api explicitly, schema evolution will take effect； Otherwise, everything goes through the original process
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
ok， will do it. thanks
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -152,7 +160,7 @@ public void scan() {
// iterate over the paths
logFormatReaderWrapper = new HoodieLogFormatReader(fs,
logFilePaths.stream().map(logFile -> new HoodieLogFile(new Path(logFile))).collect(Collectors.toList()),
- readerSchema, readBlocksLazily, reverseReader, bufferSize);
+ internalSchema != null ? null : readerSchema, readBlocksLazily, reverseReader, bufferSize);
Review comment:
when internalSchema is not null， we should forbid AbstractHoodieLogRecordScanner useing readerSchema(which is a avro Schema) to read avro record; since AbstractHoodieLogRecordScanner may not be able to read correctly data by using readerSchema.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -311,6 +319,12 @@ private boolean isNewInstantBlock(HoodieLogBlock logBlock) {
private void processDataBlock(HoodieDataBlock dataBlock) throws Exception {
// TODO (NA) - Implement getRecordItr() in HoodieAvroDataBlock and use that here
List<IndexedRecord> recs = dataBlock.getRecords();
+ if (internalSchema != null) {
+ Long currentTime = Long.parseLong(dataBlock.getLogBlockHeader().get(INSTANT_TIME));
+ InternalSchema fileSchema = TableInternalSchemaUtils.searchSchemaAndCache(currentTime, path, fs.getConf());
+ Schema readSchema = AvroSchemaUtil.convert(InternalSchemaUtils.mergeSchema(fileSchema, internalSchema), readerSchema != null ? readerSchema.getName() : "tableName");
+ recs = recs.stream().map(rec -> AvroSchemaUtil.rewriteRecord((GenericRecord) rec, readSchema)).collect(Collectors.toList());
Review comment:
yes
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
Review comment:
ok
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieUnMergedLogRecordScanner.java
##########
@@ -38,7 +38,7 @@
private HoodieUnMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
String latestInstantTime, boolean readBlocksLazily, boolean reverseReader, int bufferSize,
LogRecordScannerCallback callback, Option<InstantRange> instantRange) {
- super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false);
+ super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false, null);
Review comment:
no, spark mor/cow has not use HoodieUnmergedLogRecordScanner only hive/presto use it. this pr is only about spark adapation.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
Review comment:
ok
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
+ public MergeSchemaAction(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ this.fileSchema = fileSchema;
+ this.querySchema = querySchema;
+ this.mergeRequiredFiledForce = mergeRequiredFiledForce;
+ }
+
+ public List<Types.Field> buildNewFields(List<Types.Field> oldFields, List<Type> newTypes) {
Review comment:
done
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
Review comment:
yes， change it to protected。 this method cannot be a part of internalSchema，this method is different for different implementation classes.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
Review comment:
already done
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
+ FIRST,
+ BEFORE,
+ AFTER
+ }
+
+ private final int srcId;
+ private final int dsrId;
+ private final ColumnPositionType type;
+
+ static ColumnPositionChange first(int srcId) {
+ return new ColumnPositionChange(srcId, -1, ColumnPositionType.FIRST);
+ }
+
+ static ColumnPositionChange before(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.BEFORE);
+ }
+
+ static ColumnPositionChange after(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.AFTER);
+ }
+
+ static ColumnPositionChange get(int srcId, int dsrId, String type) {
+ switch (type) {
Review comment:
done
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChanges.java
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.SchemaChangeUtils;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class TableChanges {
Review comment:
remove helper methods to a helper class. now TableChanges is ok.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaUtil.java
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import static org.apache.avro.Schema.Type.UNION;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.action.TableChanges;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+public class AvroSchemaUtil {
Review comment:
ok
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
done,
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
done, docs added, and rename InternalSchemaParser to SerDeHelper
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ /**
+ * build a mapping from id to full field name for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to full field name
+ */
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ /**
+ * build a mapping from full field name to id for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from full field name to id
+ */
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ /**
+ * build a mapping from id to field for a internal Type.
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to field
+ */
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
Review comment:
ok
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordScanner.java
##########
@@ -45,7 +45,7 @@ private HoodieMetadataMergedLogRecordScanner(FileSystem fs, String basePath, Lis
String spillableMapBasePath, Set<String> mergeKeyFilter,
ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled) {
super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, false, false, bufferSize,
- spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false);
+ spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false, null);
Review comment:
spark mor/cow has not use HoodieUnmergedLogRecordScanner only hive/presto use it. this pr is only about spark adapation.
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala
##########
@@ -52,26 +54,33 @@ case class HoodieMergeOnReadTableState(tableStructSchema: StructType,
requiredAvroSchema: String,
hoodieRealtimeFileSplits: List[HoodieMergeOnReadFileSplit],
preCombineField: Option[String],
- recordKeyFieldOpt: Option[String])
+ recordKeyFieldOpt: Option[String],
+ internalSchema: Option[InternalSchema] = None,
+ requiredInternalSchema: Option[InternalSchema] = None)
class MergeOnReadSnapshotRelation(val sqlContext: SQLContext,
val optParams: Map[String, String],
val userSchema: StructType,
val globPaths: Option[Seq[Path]],
val metaClient: HoodieTableMetaClient)
- extends BaseRelation with PrunedFilteredScan with Logging {
+ extends BaseRelation with PrunedFilteredScan with Logging with SparkAdapterSupport {
private val conf = sqlContext.sparkContext.hadoopConfiguration
private val jobConf = new JobConf(conf)
// use schema from latest metadata, if not present, read schema from the data file
private val schemaUtil = new TableSchemaResolver(metaClient)
- private lazy val tableAvroSchema = {
+ private lazy val (tableAvroSchema, internalSchema) = {
try {
- schemaUtil.getTableAvroSchema
+ val internalSchemaOpt = schemaUtil.getTableInternalSchemaFromCommitMetadata
+ if (internalSchemaOpt.isPresent) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-924829257
@bvaradar . thanks for your review。 I will try to solve these problems。 There is a little question, do we need to add all adaptations to the spark engine on this pr。 if needed，full adaptation of spark engine（both mor and cow） will be added、
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r711553129
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Internal schema for hudi table.
+ * used to support schema evolution.
+ */
+public class InternalSchema implements Serializable {
+
+ private static final long DEFAULT_VERSION_ID = 0;
+
+ private final RecordType record;
+
+ private int maxColumnId;
+ private long versionId;
+
+ private transient Map<Integer, Field> idToField = null;
+ private transient Map<String, Integer> nameToId = null;
+ private transient Map<Integer, String> idToName = null;
+
+ public InternalSchema(List columns) {
+ this(DEFAULT_VERSION_ID, columns);
+ }
+
+ public InternalSchema(Field... columns) {
+ this(DEFAULT_VERSION_ID, Arrays.asList(columns));
+ }
+
+ public InternalSchema(long versionId, List<Field> cols) {
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ maxColumnId = idToName.keySet().stream().max(Comparator.comparing(Integer::valueOf)).get();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, List<Field> cols) {
+ this.maxColumnId = maxColumnId;
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, Field... cols) {
+ this(versionId, maxColumnId, Arrays.asList(cols));
+ }
+
+ public RecordType getRecord() {
+ return record;
+ }
+
+ private Map<Integer, String> buildIdToName() {
+ if (idToName == null) {
+ idToName = InternalSchemaUtils.buildIdToName(record);
+ }
+ return idToName;
+ }
+
+ private Map<String, Integer> buildNameToId() {
+ if (nameToId == null) {
+ nameToId = InternalSchemaUtils.buildNameToId(record);
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba407cc5af64d50eaa18f5df4adf1307853c264d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251)
* 1a0b7d831b32d1704c1837a2df77bfe4d36724cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-930001505
@bvaradar @codope @leesf . could you pls help me review this pr again, thanks
code changes
1) support mor(incremental/realtime/optimize) read/write
2) support cow (incremental/realtime) read/write
3) support spark3 DDL. include:
alter statement:
* ALTER TABLE table1 ALTER COLUMN a.b.c TYPE bigint
* ALTER TABLE table1 ALTER COLUMN a.b.c SET NOT NULL
* ALTER TABLE table1 ALTER COLUMN a.b.c DROP NOT NULL
* ALTER TABLE table1 ALTER COLUMN a.b.c COMMENT 'new comment'
* ALTER TABLE table1 ALTER COLUMN a.b.c FIRST
* ALTER TABLE table1 ALTER COLUMN a.b.c AFTER x
add statement:
* ALTER TABLE table1 ADD COLUMNS (col_name data_type [COMMENT col_comment], ...);
rename:
* ALTER TABLE table1 RENAME COLUMN a.b.c TO x
drop:
* ALTER TABLE table1 DROP COLUMN a.b.c
* ALTER TABLE table1 DROP COLUMNS a.b.c, x, y
set/unset Properties:
* ALTER TABLE table SET TBLPROPERTIES ('table_property' = 'property_value');
* ALTER TABLE table UNSET TBLPROPERTIES [IF EXISTS] ('comment', 'key');
4) support spark2 DDL.
5) add FileBaseInternalSchemasManager to manger history schemas, and save historySchemas in "./hoodie/.schema" . now we no need to save historySchemas into commit file.
6) add segment lock to TableInternalSchemaUtils to support concurrent read and write cache.
7) rename mergeSchemaAction to SchemaMerger; remove helper methods from TableChanges to a helper class, now TableChanges is ok; use visitor mode to produce nameToId for internalSchema; and other samll fixed.
Remaining problem: add more UT for this pr, add support for bootstrap table.
@bvaradar forgive me this change is too large, i still use squahsing. Subsequent modifications will be in the form of add commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715455567
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaf33e3a28680ab5febca7df70937ce543619a94 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632)
* 8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Support full Schema evolution for Spark/Hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-1050840323
close this pr, as this pr is too old, and we also have a new pr about this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r711538619
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Internal schema for hudi table.
+ * used to support schema evolution.
+ */
+public class InternalSchema implements Serializable {
+
+ private static final long DEFAULT_VERSION_ID = 0;
+
+ private final RecordType record;
+
+ private int maxColumnId;
+ private long versionId;
+
+ private transient Map<Integer, Field> idToField = null;
Review comment:
no problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709166519
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
Review comment:
for nested field, just use `a.b.c` ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
Review comment:
for nested field, just use the fullname like `a.b.c` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710897845
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
Review comment:
yes i will add docs。 sorry i cannot understand what's the mean of requireNonNull,
could you explain it in detail for me? thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919918921
This PR is about RFC-33? Have we reached an agreement on the design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709162430
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
Review comment:
what does parentName means here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 42ca73a97f00023736feeda07e96d03e7a621c9b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459)
* 0fe30385aaabcc0b7f1474228d397ea67e174804 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-930001505
@bvaradar @codope @leesf . could you pls help me review this pr again, thanks
code changes
1) support mor(incremental/realtime/optimize) read/write
2) support cow (incremental/realtime) read/write
3) support spark3 DDL. include:
alter statement:
* ALTER TABLE table1 ALTER COLUMN a.b.c TYPE bigint
* ALTER TABLE table1 ALTER COLUMN a.b.c SET NOT NULL
* ALTER TABLE table1 ALTER COLUMN a.b.c DROP NOT NULL
* ALTER TABLE table1 ALTER COLUMN a.b.c COMMENT 'new comment'
* ALTER TABLE table1 ALTER COLUMN a.b.c FIRST
* ALTER TABLE table1 ALTER COLUMN a.b.c AFTER x
add statement:
* ALTER TABLE table1 ADD COLUMNS (col_name data_type [COMMENT col_comment], ...);
rename:
* ALTER TABLE table1 RENAME COLUMN a.b.c TO x
drop:
* ALTER TABLE table1 DROP COLUMN a.b.c
* ALTER TABLE table1 DROP COLUMNS a.b.c, x, y
set/unset Properties:
* ALTER TABLE table SET TBLPROPERTIES ('table_property' = 'property_value');
* ALTER TABLE table UNSET TBLPROPERTIES [IF EXISTS] ('comment', 'key');
4) support spark2 DDL.
5) add FileBaseInternalSchemasManager to manger history schemas, and save historySchemas in "./hoodie/.schema" . now we no need to save historySchemas into commit file.
6) add segment lock to TableInternalSchemaUtils to support concurrent read and write cache.
7) rename mergeSchemaAction to SchemaMerger; remove helper methods from TableChanges to a helper class, now TableChanges is ok; use visitor mode to produce nameToId for internalSchema; and other samll fixed.
Remaining problem: add more UT for this pr, add support for bootstrap table.
@bvaradar forgive me this change is too large, i still use squahsing. Subsequent modifications will be in the form of add commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-961588408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-928824868
@xiarixiaoyao : Can you add commits to this PR instead of squashing. It makes things easy for us to find the delta changes. We can do final squash before landing the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717230330
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
@xiarixiaoyao : The synchronous metadata table support is still in the works and is getting ready. Its better to decouple both the projects and at the same time avoid adding historical schema to commit files. As an interim measure, lets write as separate files under .hoodie/schema and then we can do a follow up PR whenever synchronous metadata table is done. cc @vinothchandar @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717275228
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
+ FIRST,
+ BEFORE,
+ AFTER
+ }
+
+ private final int srcId;
+ private final int dsrId;
+ private final ColumnPositionType type;
+
+ static ColumnPositionChange first(int srcId) {
+ return new ColumnPositionChange(srcId, -1, ColumnPositionType.FIRST);
+ }
+
+ static ColumnPositionChange before(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.BEFORE);
+ }
+
+ static ColumnPositionChange after(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.AFTER);
+ }
+
+ static ColumnPositionChange get(int srcId, int dsrId, String type) {
+ switch (type) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717192565
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
> We need to introduce a config to enable/disable schema evolution. Only if the config is enables, we should let any side-effects of schema evolution to take effect.
may no need introduce a config. only when user execute alter SQL /alter api explicitly, schema evolution will take effect； Otherwise, everything goes through the original process
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717270850
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
+ public MergeSchemaAction(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ this.fileSchema = fileSchema;
+ this.querySchema = querySchema;
+ this.mergeRequiredFiledForce = mergeRequiredFiledForce;
+ }
+
+ public List<Types.Field> buildNewFields(List<Types.Field> oldFields, List<Type> newTypes) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717274933
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
Review comment:
yes， change it to protected。 this method cannot be a part of internalSchema，this method is different for different implementation classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0fe30385aaabcc0b7f1474228d397ea67e174804 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460)
* a423a63a9a530365b531a4199a5010dd52708d86 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717261334
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -311,6 +319,12 @@ private boolean isNewInstantBlock(HoodieLogBlock logBlock) {
private void processDataBlock(HoodieDataBlock dataBlock) throws Exception {
// TODO (NA) - Implement getRecordItr() in HoodieAvroDataBlock and use that here
List<IndexedRecord> recs = dataBlock.getRecords();
+ if (internalSchema != null) {
+ Long currentTime = Long.parseLong(dataBlock.getLogBlockHeader().get(INSTANT_TIME));
+ InternalSchema fileSchema = TableInternalSchemaUtils.searchSchemaAndCache(currentTime, path, fs.getConf());
+ Schema readSchema = AvroSchemaUtil.convert(InternalSchemaUtils.mergeSchema(fileSchema, internalSchema), readerSchema != null ? readerSchema.getName() : "tableName");
+ recs = recs.stream().map(rec -> AvroSchemaUtil.rewriteRecord((GenericRecord) rec, readSchema)).collect(Collectors.toList());
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709815386
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574)
* cdcf299c9f4e743cd74d9266603a30bf3301c3b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-939724336
@bvaradar rebased the code and add config to control schema evolution。 the test fail has no relate to this pr。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717260982
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -152,7 +160,7 @@ public void scan() {
// iterate over the paths
logFormatReaderWrapper = new HoodieLogFormatReader(fs,
logFilePaths.stream().map(logFile -> new HoodieLogFile(new Path(logFile))).collect(Collectors.toList()),
- readerSchema, readBlocksLazily, reverseReader, bufferSize);
+ internalSchema != null ? null : readerSchema, readBlocksLazily, reverseReader, bufferSize);
Review comment:
when internalSchema is not null， we should forbid AbstractHoodieLogRecordScanner useing readerSchema(which is a avro Schema) to read avro record; since AbstractHoodieLogRecordScanner may not be able to read correctly data by using readerSchema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710866801
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+/**
+ * helper class.
+ * use static cache to cache historySchema info to prevent fetching from hdfs every time.
+ * all task belong to the same executor/container will share this cache.
+ *
+ */
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
+ HISTORICAL_SCHEMA_CACHE = Caffeine.newBuilder().maximumSize(1000).weakValues().build();
+
+ public static InternalSchema searchSchemaAndCache(long versionID, String tablePath, Configuration hadoopConf) {
+ TreeMap<Long, InternalSchema> historicalSchemas = HISTORICAL_SCHEMA_CACHE.getIfPresent(tablePath);
+ if (historicalSchemas == null || InternalSchemaParser.searchSchema(versionID, historicalSchemas) == null) {
+ historicalSchemas = getHistoricalSchemas(tablePath, hadoopConf);
+ HISTORICAL_SCHEMA_CACHE.put(tablePath, historicalSchemas);
+ } else {
+ long maxVersionId = historicalSchemas.keySet().stream().max(Long::compareTo).get();
Review comment:
good suggestion， thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c0f0a8e1025051d6077e168b4ca15f4ee8edccc1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2772",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
}, {
"hash" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773",
"triggerID" : "fc5223c3a568ecec2ac1f1ba26815a0682c56d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc5223c3a568ecec2ac1f1ba26815a0682c56d47 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 101b85af0521d578a9a42eb50a8b523bc65d0abf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249)
* ba407cc5af64d50eaa18f5df4adf1307853c264d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64c7ecf585993377b6a0d9b18e14003068471e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710873627
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -87,4 +88,9 @@ trait SparkAdapter extends Serializable {
* Create Like expression.
*/
def createLike(left: Expression, right: Expression): Expression
+
+ /**
+ * Create parquet file read for hudi.
+ */
+ def createHoodieParquetFileFormat(): ParquetFileFormat
Review comment:
yes，we need do that。but i think this pr is more like the prototype of id-schema, I hope we can reach a basic agreement on this pr。 full spark adaption will be another pr 。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao closed pull request #3668: [RFC-33] [HUDI-2429][WIP] Support full Schema evolution for Spark/Hive
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao closed pull request #3668:
URL: https://github.com/apache/hudi/pull/3668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710891920
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
+ Integer nameId = querySchema.findIdByName(name);
+ if (nameId == null) {
+ throw new IllegalArgumentException(String.format("cannot found filter col name：%s from querySchema: %s", name, querySchema));
+ }
+ if (fileSchema.findField(nameId) == null) {
+ // added operation found
+ // the read file does not contain current col, so current colFilter is invalid
+ return "";
+ } else {
+ if (name.equals(fileSchema.findfullName(nameId))) {
Review comment:
Yes, it can be done, but the algorithm is poorly readable. i tread to not simplified those code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709168436
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumns(String colName, Schema schema) {
+ addColumns(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(String colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ delete.deleteColumn(colName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
+ * @param newName new name for current col. no need to specify fullName.
+ */
+ public void renameColumn(String colName, String newName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.renameColumn(colName, newName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col type for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param nullable .
+ */
+ public void updateColumnNullability(String colName, boolean nullable) {
Review comment:
this method is used to alter table column type such as from int to long?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709186039
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
+ HISTORICAL_SCHEMA_CACHE = Caffeine.newBuilder().maximumSize(1000).weakValues().build();
+
+ public static InternalSchema searchSchemaAndCache(long versionID, String tablePath, Configuration hadoopConf) {
Review comment:
please add some docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2d8b3f67e3556ccb84309b381ea513731537f8b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285)
* b2d8b3f67e3556ccb84309b381ea513731537f8b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64c7ecf585993377b6a0d9b18e14003068471e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235)
* 101b85af0521d578a9a42eb50a8b523bc65d0abf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-928824868
@xiarixiaoyao : Can you add commits to this PR instead of squashing. It makes things easy for us to find the delta changes. We can do final squash before landing the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717261469
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1a0b7d831b32d1704c1837a2df77bfe4d36724cc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263)
* 5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2632",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
}, {
"hash" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737",
"triggerID" : "8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c",
"triggerType" : "PUSH"
}, {
"hash" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8588a4deeb8fa60fa18d6b9110e7944ad77b8d9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2737)
* d1c9d35ed6768a2411ce597ba5366d9f9e1ac82b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-940584211
@codope thanks for your try this pr. notice that we should not use --conf "hoodie.schema.evolution.enable=true", this conf is not start with spark/hadoop/hive spark will ignore this conf. see the test case, we can execute set command to enable schema evolution, sql("set hoodie.schema.evolution.enable=true"). another things it's very strange that rename cannot work well, TestSpark3DDL already contains rename example, spark2 has no same test case . maybe we can discuss this problem by slack, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a423a63a9a530365b531a4199a5010dd52708d86 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466)
* 4af82a25703eca83ff1e6e8d12055bcfec8357e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
}, {
"hash" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aaf33e3a28680ab5febca7df70937ce543619a94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89dab78876b2512aa4967ced70da27f6fdb46b14 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619)
* aaf33e3a28680ab5febca7df70937ce543619a94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-939599746
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cdcf299c9f4e743cd74d9266603a30bf3301c3b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cdcf299c9f4e743cd74d9266603a30bf3301c3b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576)
* 89dab78876b2512aa4967ced70da27f6fdb46b14 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-921467046
@codope thanks for your review. Let me answer some your questions first，pls forgive me for being busy today, I need to modify something for the pr of zorder
**Let's add docs for all the public classes and APIs.**
ok，i will added。
**Does the merge schema action handle evolution of non-leaf fields in nested fields? For example, if a.b.c is renamed to a.d.c.**
this is a strange demand， if you want to change a.b.c to a.d.c , why donot use spark.sql(s"alter table ${tableName} rename column a.b to d"). we support handle all the non-leaf fields.
**IIUC, the patch has not yet handled old or existing schema compatibility as mentioned in the RFC right?**
already deal with this situation. pls see the TestSpark3DDL, we first create a table and insert some data to it. now no id-schema is produced. then we do schema change, and the id-schema is produced. the test result confirmed this。
**Since schema history is being changed not only at the write time but also at the read time, so we need to think of both writer and reader concurrency.**
A little doubtful， hudi is snapshot isolated， maybe only need to deal with concurrency between write and write， if i am wrong pls fix me ， thanks
Now there is the most important question， I now tend to use metatable to store historical schema。 as we know metatable use hfile to store data， the hfile has a very good point query performance， what do you suggest？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709817580
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
Review comment:
Suppose we have a nested column. the parentName of a.b.c is a.b; the parentName of a.b is a; the parentName of a is ""
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r715470600
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkSchemaUtils.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.ArrayType$;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.BooleanType$;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.CharType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DecimalType$;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.DoubleType$;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.FloatType$;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.MapType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.sql.types.TimestampType$;
+import org.apache.spark.sql.types.UserDefinedType;
+import org.apache.spark.sql.types.VarcharType;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class SparkSchemaUtils {
+ private SparkSchemaUtils() {
+
+ }
+
+ public static final String HOODIE_QUERY_SCHEMA = "hadoop.hoodie.querySchema";
+ public static final String HOODIE_TABLE_PATH = "hadoop.hoodie.tablePath";
+
+ /**
+ * Converts a spark schema to an hudi internal schema. Fields without IDs are kept and assigned fallback IDs.
+ *
+ * @param sparkSchema a spark schema
+ * @return a matching internal schema for the provided spark schema
+ */
+ public static InternalSchema convertStructTypeToInternalSchema(StructType sparkSchema) {
+ Type newType = buildTypeFromStructType(sparkSchema, true, new AtomicInteger(0));
+ return new InternalSchema(((Types.RecordType)newType).fields());
+ }
+
+ public static Type buildTypeFromStructType(DataType sparkType, Boolean firstVisitRoot, AtomicInteger nextId) {
Review comment:
no, we just need a class which wrapper the increment id。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r713621765
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java
##########
@@ -493,4 +496,83 @@ public static MessageType readSchemaFromLogFile(FileSystem fs, Path path) throws
}
return null;
}
+
+ /**
+ * Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
+ *
+ * @return InternalSchema for this table
+ */
+ public Option<InternalSchema> getTableInternalSchemaFromCommitMetadata() {
+ HoodieTimeline timeline = metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+ if (timeline.lastInstant().isPresent()) {
+ return getTableInternalSchemaFromCommitMetadata(timeline.lastInstant().get());
+ } else {
+ return Option.empty();
+ }
+ }
+
+ /**
+ * Gets the InternalSchema for a hoodie table from the HoodieCommitMetadata of the instant.
+ *
+ * @return InternalSchema for this table
+ */
+ private Option<InternalSchema> getTableInternalSchemaFromCommitMetadata(HoodieInstant instant) {
+ try {
+ HoodieTimeline timeline = metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+ byte[] data = timeline.getInstantDetails(instant).get();
Review comment:
instead of timeline, use metadata.getActiveTimeline()
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
Review comment:
when will this case happen ? SCHEMA_KEY is passed but not the LATESTSCHEMA ?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkSchemaUtils.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.ArrayType$;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.BooleanType$;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.CharType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DecimalType$;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.DoubleType$;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.FloatType$;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.MapType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.sql.types.TimestampType$;
+import org.apache.spark.sql.types.UserDefinedType;
+import org.apache.spark.sql.types.VarcharType;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class SparkSchemaUtils {
+ private SparkSchemaUtils() {
+
+ }
+
+ public static final String HOODIE_QUERY_SCHEMA = "hadoop.hoodie.querySchema";
+ public static final String HOODIE_TABLE_PATH = "hadoop.hoodie.tablePath";
+
+ /**
+ * Converts a spark schema to an hudi internal schema. Fields without IDs are kept and assigned fallback IDs.
+ *
+ * @param sparkSchema a spark schema
+ * @return a matching internal schema for the provided spark schema
+ */
+ public static InternalSchema convertStructTypeToInternalSchema(StructType sparkSchema) {
+ Type newType = buildTypeFromStructType(sparkSchema, true, new AtomicInteger(0));
+ return new InternalSchema(((Types.RecordType)newType).fields());
+ }
+
+ public static Type buildTypeFromStructType(DataType sparkType, Boolean firstVisitRoot, AtomicInteger nextId) {
Review comment:
why AtomicInteger ? Is this method expected to be called concurrently ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -152,7 +160,7 @@ public void scan() {
// iterate over the paths
logFormatReaderWrapper = new HoodieLogFormatReader(fs,
logFilePaths.stream().map(logFile -> new HoodieLogFile(new Path(logFile))).collect(Collectors.toList()),
- readerSchema, readBlocksLazily, reverseReader, bufferSize);
+ internalSchema != null ? null : readerSchema, readBlocksLazily, reverseReader, bufferSize);
Review comment:
not following this. why this is set null when internalSchema is not null ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChanges.java
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.SchemaChangeUtils;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class TableChanges {
Review comment:
This class contains a mix of helper methods and derived classes of TableChange. Lets rename this class as TableSchemaHelper.
There are bunch of helper classes across. Will need to rethink alignment to see if we can make readability better. Will focus on it in the next pass.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
Review comment:
nit: Rename to SchemaMerger ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
Review comment:
Also, can you please add some comments on the if blocks as there are quite a few of them.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
We need to introduce a config to enable/disable schema evolution. Only if the config is enables, we should let any side-effects of schema evolution to take effect.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
Review comment:
Similar logic needs to be also be present in BaseCommitExecutor.commit.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1100,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumn(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumn(String colName, Schema schema) {
+ addColumn(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(List<String> colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ colName.stream().forEach(col -> delete.deleteColumn(col));
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
+ * @param newName new name for current col. no need to specify fullName.
+ */
+ public void renameColumn(String colName, String newName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.renameColumn(colName, newName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col nullable attribute for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param nullable .
+ */
+ public void updateColumnNullability(String colName, boolean nullable) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.updateColumnNullability(colName, nullable);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col comment for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param doc .
+ */
+ public void updateColumnComment(String colName, String doc) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.updateColumnComment(colName, doc);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * reorder the position of col.
+ *
+ * @param colName column which need to be reordered. if we want to change col from a nested filed, the fullName should be specify.
+ * @param referColName reference position.
+ * @param orderType col position change type. now support three change types: first/after/before
+ */
+ public void reOrderColPosition(String colName, String referColName, String orderType) {
+ if (colName == null || orderType == null || referColName == null) {
Review comment:
For all the schema manipulation APIs in this class, can you move the schema changing logic to separate helper class and this class can simply delegate, getback the schema(s) to be persisted and commits.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkSchemaUtils.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.spark.sql.types.ArrayType;
+import org.apache.spark.sql.types.ArrayType$;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.BooleanType$;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.CharType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DateType$;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DecimalType$;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.DoubleType$;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.FloatType$;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.IntegerType$;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.MapType;
+import org.apache.spark.sql.types.MapType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.sql.types.TimestampType$;
+import org.apache.spark.sql.types.UserDefinedType;
+import org.apache.spark.sql.types.VarcharType;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class SparkSchemaUtils {
+ private SparkSchemaUtils() {
+
+ }
+
+ public static final String HOODIE_QUERY_SCHEMA = "hadoop.hoodie.querySchema";
Review comment:
what are the purpose of this config and the below one ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1100,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumn(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
Review comment:
rename add to addCol
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1100,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
Review comment:
positionChangeType should be enum
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1100,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumn(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
+ String referParentName = TableChanges.getParentName(position);
+ if (!parentName.equals(referParentName)) {
+ throw new IllegalArgumentException("cannot reorder two columns which has different parent");
+ }
+ add.addPositionChange(colName, position, positionType);
+ } else if (positionType != null && positionType.equals("first")) {
+ add.addPositionChange(colName, "", "first");
+ }
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), add);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ public void addColumn(String colName, Schema schema) {
+ addColumn(colName, schema, null, null, null);
+ }
+
+ /**
+ * delete columns to table.
+ *
+ * @param colName col name to be deleted. if we want to delete col from a nested filed, the fullName should be specify
+ */
+ public void deleteColumns(List<String> colName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnDeleteChange delete = TableChanges.ColumnDeleteChange.get(pair.getLeft());
+ colName.stream().forEach(col -> delete.deleteColumn(col));
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), delete);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * rename col name for hudi table.
+ *
+ * @param colName col name to be renamed. if we want to rename col from a nested filed, the fullName should be specify
+ * @param newName new name for current col. no need to specify fullName.
+ */
+ public void renameColumn(String colName, String newName) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.renameColumn(colName, newName);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
+ commitTableChange(newSchema, pair.getRight());
+ }
+
+ /**
+ * update col nullable attribute for hudi table.
+ *
+ * @param colName col name to be changed. if we want to change col from a nested filed, the fullName should be specify
+ * @param nullable .
+ */
+ public void updateColumnNullability(String colName, boolean nullable) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnUpdateChange updateChange = TableChanges.ColumnUpdateChange.get(pair.getLeft());
+ updateChange.updateColumnNullability(colName, nullable);
+ InternalSchema newSchema = SchemaChangeUtils.applyTableChanges2Schema(pair.getLeft(), updateChange);
Review comment:
nit: applyTableChanges2Schema to simply apply ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -213,6 +224,27 @@ protected void commit(HoodieTable table, String commitActionType, String instant
List<HoodieWriteStat> stats) throws IOException {
LOG.info("Committing " + instantTime + " action " + commitActionType);
HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+ // do save internal schema to support Implicitly add columns in write process
+ if (!metadata.getExtraMetadata().containsKey(InternalSchemaParser.LATESTSCHEMA) && metadata.getExtraMetadata().containsKey(SCHEMA_KEY)) {
+ TableSchemaResolver schemaUtils = new TableSchemaResolver(table.getMetaClient());
+ String historySchemaStr = schemaUtils.getTableHistorySchemaStrFromCommitMetadata().orElse("");
+ if (!historySchemaStr.isEmpty()) {
Review comment:
High level : Instead of storing historical schema in commit file, its better to store them as separate files under a new directory under .hoodie. Something like .hoodie/schema/...
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+/**
+ * helper class.
+ * use static cache to cache historySchema info to prevent fetching from hdfs every time.
+ * all task belong to the same executor/container will share this cache.
+ *
+ */
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
Review comment:
a static cache is problematic. Query engines will reuse the same JVM instance for running different queries. Lets avoid this pattern.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieUnMergedLogRecordScanner.java
##########
@@ -38,7 +38,7 @@
private HoodieUnMergedLogRecordScanner(FileSystem fs, String basePath, List<String> logFilePaths, Schema readerSchema,
String latestInstantTime, boolean readBlocksLazily, boolean reverseReader, int bufferSize,
LogRecordScannerCallback callback, Option<InstantRange> instantRange) {
- super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false);
+ super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, readBlocksLazily, reverseReader, bufferSize, instantRange, false, null);
Review comment:
Is this a todo ? Shouldn't we be passing internalSchema ?
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -87,4 +88,9 @@ trait SparkAdapter extends Serializable {
* Create Like expression.
*/
def createLike(left: Expression, right: Expression): Expression
+
+ /**
+ * Create parquet file read for hudi.
+ */
+ def createHoodieParquetFileFormat(): ParquetFileFormat
Review comment:
@xiarixiaoyao : It would be helpful to review if you can list the gaps in the current PR before reaching full spark support
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ /**
+ * build a mapping from id to full field name for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to full field name
+ */
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ /**
+ * build a mapping from full field name to id for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from full field name to id
+ */
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ /**
+ * build a mapping from id to field for a internal Type.
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to field
+ */
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
Review comment:
Consider using Visitor design pattern using classes instead of static methods.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
Review comment:
Should this be protected ? From the description and name, this method should be part of the InternalSchema itself and not as part of ColumnChange.
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMergedLogRecordScanner.java
##########
@@ -45,7 +45,7 @@ private HoodieMetadataMergedLogRecordScanner(FileSystem fs, String basePath, Lis
String spillableMapBasePath, Set<String> mergeKeyFilter,
ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled) {
super(fs, basePath, logFilePaths, readerSchema, latestInstantTime, maxMemorySizeInBytes, false, false, bufferSize,
- spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false);
+ spillableMapBasePath, Option.empty(), false, diskMapType, isBitCaskDiskMapCompressionEnabled, false, null);
Review comment:
Same comment as the one in HoodieUnMergedLogRecordScanner
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
+ public MergeSchemaAction(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ this.fileSchema = fileSchema;
+ this.querySchema = querySchema;
+ this.mergeRequiredFiledForce = mergeRequiredFiledForce;
+ }
+
+ public List<Types.Field> buildNewFields(List<Types.Field> oldFields, List<Type> newTypes) {
Review comment:
Looks like this called for Record types, can you rename the methods to convey full meaning. It would be great if you could look at the naming across the code changes to make sure they are consistent.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
Add docs to all public APIs
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
Review comment:
these enums must be reused in AbstractHoodieWriteClient
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala
##########
@@ -52,26 +54,33 @@ case class HoodieMergeOnReadTableState(tableStructSchema: StructType,
requiredAvroSchema: String,
hoodieRealtimeFileSplits: List[HoodieMergeOnReadFileSplit],
preCombineField: Option[String],
- recordKeyFieldOpt: Option[String])
+ recordKeyFieldOpt: Option[String],
+ internalSchema: Option[InternalSchema] = None,
+ requiredInternalSchema: Option[InternalSchema] = None)
class MergeOnReadSnapshotRelation(val sqlContext: SQLContext,
val optParams: Map[String, String],
val userSchema: StructType,
val globPaths: Option[Seq[Path]],
val metaClient: HoodieTableMetaClient)
- extends BaseRelation with PrunedFilteredScan with Logging {
+ extends BaseRelation with PrunedFilteredScan with Logging with SparkAdapterSupport {
private val conf = sqlContext.sparkContext.hadoopConfiguration
private val jobConf = new JobConf(conf)
// use schema from latest metadata, if not present, read schema from the data file
private val schemaUtil = new TableSchemaResolver(metaClient)
- private lazy val tableAvroSchema = {
+ private lazy val (tableAvroSchema, internalSchema) = {
try {
- schemaUtil.getTableAvroSchema
+ val internalSchemaOpt = schemaUtil.getTableInternalSchemaFromCommitMetadata
+ if (internalSchemaOpt.isPresent) {
Review comment:
nit: You can simplify with something like
schemaUtil.getTableAvroSchema, internalSchemaOpt.orElse(null)
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaUtil.java
##########
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import static org.apache.avro.Schema.Type.UNION;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.action.TableChanges;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+public class AvroSchemaUtil {
Review comment:
For the many conversions we are doing Avro, InternalSchema, Spark, Parquet... , instead of implementing these methods in helper/utils class, let rename the classes as "xxxxConverter" and make sure only the conversion functions are defined.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/MergeSchemaAction.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * auxiliary class.
+ * help to merge file schema and query schema to produce final read schema for avro/parquet file
+ */
+public class MergeSchemaAction {
+ private final InternalSchema fileSchema;
+ private final InternalSchema querySchema;
+ private final Boolean mergeRequiredFiledForce;
Review comment:
type. Can you add comments when this flag is turned on ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/TableChange.java
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+
+/**
+ * TableChange subclasses represent requested changes to a table.
+ * now only column changes support.
+ * to do support partition changes
+ */
+public interface TableChange {
+ /* The action Type of schema change. */
+ enum ColumnChangeID {
+ ADD, UPDATE, DELETE, PROPERTY_CHANGE;
+ private String name;
+
+ private ColumnChangeID() {
+ this.name = this.name().toLowerCase(Locale.ROOT);
+ }
+
+ public String getName() {
+ return name;
+ }
+ }
+
+ static ColumnChangeID fromValue(String value) {
+ switch (value.toLowerCase(Locale.ROOT)) {
+ case "add":
+ return ColumnChangeID.ADD;
+ case "change":
+ return ColumnChangeID.UPDATE;
+ case "delete":
+ return ColumnChangeID.DELETE;
+ case "property":
+ return ColumnChangeID.PROPERTY_CHANGE;
+ default:
+ throw new IllegalArgumentException("Invalid value of Type.");
+ }
+ }
+
+ ColumnChangeID columnChangeId();
+
+ default boolean withPositionChange() {
+ return false;
+ }
+
+ abstract class BaseColumnChange implements TableChange {
+ protected final InternalSchema internalSchema;
+ protected final Map<Integer, Integer> id2parent;
+ protected final Map<Integer, ArrayList<ColumnPositionChange>> positionChangeMap = new HashMap<>();
+
+ BaseColumnChange(InternalSchema schema) {
+ this.internalSchema = schema;
+ this.id2parent = InternalSchemaUtils.index2Parents(schema.getRecord());
+ }
+
+ /**
+ * add position change.
+ *
+ * @param srcName column which need to be reordered
+ * @param dsrName reference position
+ * @param orderType change types
+ * @return this
+ */
+ public BaseColumnChange addPositionChange(String srcName, String dsrName, String orderType) {
+ Integer srcId = findIdByFullName(srcName);
+ Integer dsrId = findIdByFullName(dsrName);
+ Integer srcParentId = id2parent.get(srcId);
+ Integer dsrParentId = id2parent.get(dsrId);
+ int parentId;
+ if (srcParentId != null && dsrParentId != null && srcParentId.equals(dsrParentId)) {
+ Types.Field parentField = internalSchema.findField(srcParentId);
+ if (!(parentField.type() instanceof Types.RecordType)) {
+ throw new HoodieSchemaException(String.format("only support reorder fields in struct type, but find: %s", parentField.type()));
+ }
+ parentId = parentField.fieldId();
+ } else if (srcParentId == null && dsrParentId == null) {
+ parentId = -1;
+ } else if (srcParentId != null && dsrParentId == null && orderType.equals("first")) {
+ parentId = srcParentId;
+ } else {
+ throw new HoodieSchemaException("cannot order position from different parent");
+ }
+
+ ArrayList<ColumnPositionChange> changes = positionChangeMap.getOrDefault(parentId, new ArrayList<>());
+ changes.add(ColumnPositionChange.get(srcId, dsrId, orderType));
+ positionChangeMap.put(parentId, changes);
+ return this;
+ }
+
+ /**
+ * abstract method.
+ * give a column fullName and return the field id
+ *
+ * @param fullName column fullName
+ * @return field id of current column
+ */
+ public abstract Integer findIdByFullName(String fullName);
+
+ @Override
+ public boolean withPositionChange() {
+ return false;
+ }
+ }
+
+ /**
+ * Column position change.
+ * now support three change types: FIRST/AFTER/BEFORE
+ * FIRST means the specified column should be the first column.
+ * AFTER means the specified column should be put after the given column.
+ * BEFORE means the specified column should be put before the given column.
+ * Note that, the specified column may be a nested field:
+ * AFTER/BEFORE means the given columns should in the same struct;
+ * FIRST means this field should be the first one within the struct.
+ */
+ class ColumnPositionChange {
+ public enum ColumnPositionType {
+ FIRST,
+ BEFORE,
+ AFTER
+ }
+
+ private final int srcId;
+ private final int dsrId;
+ private final ColumnPositionType type;
+
+ static ColumnPositionChange first(int srcId) {
+ return new ColumnPositionChange(srcId, -1, ColumnPositionType.FIRST);
+ }
+
+ static ColumnPositionChange before(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.BEFORE);
+ }
+
+ static ColumnPositionChange after(int srcId, int dsrId) {
+ return new ColumnPositionChange(srcId, dsrId, ColumnPositionType.AFTER);
+ }
+
+ static ColumnPositionChange get(int srcId, int dsrId, String type) {
+ switch (type) {
Review comment:
Can we use the enum type here ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
Review comment:
!position.isEmpty() is still appearing twice. Can you take a look ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
looks like this class performs both serialization and deserialization. If that is the case, lets move those methods to a SerDeHelper class
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/AbstractHoodieLogRecordScanner.java
##########
@@ -311,6 +319,12 @@ private boolean isNewInstantBlock(HoodieLogBlock logBlock) {
private void processDataBlock(HoodieDataBlock dataBlock) throws Exception {
// TODO (NA) - Implement getRecordItr() in HoodieAvroDataBlock and use that here
List<IndexedRecord> recs = dataBlock.getRecords();
+ if (internalSchema != null) {
+ Long currentTime = Long.parseLong(dataBlock.getLogBlockHeader().get(INSTANT_TIME));
+ InternalSchema fileSchema = TableInternalSchemaUtils.searchSchemaAndCache(currentTime, path, fs.getConf());
+ Schema readSchema = AvroSchemaUtil.convert(InternalSchemaUtils.mergeSchema(fileSchema, internalSchema), readerSchema != null ? readerSchema.getName() : "tableName");
+ recs = recs.stream().map(rec -> AvroSchemaUtil.rewriteRecord((GenericRecord) rec, readSchema)).collect(Collectors.toList());
Review comment:
We need to rewrite record only when there is some delta in the schema. Can we make sure that is done ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b2d8b3f67e3556ccb84309b381ea513731537f8b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400)
* b5e0b228840b7c0575b6eaaf888e0f5bb6024b91 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2251",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
}, {
"hash" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2263",
"triggerID" : "1a0b7d831b32d1704c1837a2df77bfe4d36724cc",
"triggerType" : "PUSH"
}, {
"hash" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2285",
"triggerID" : "5c9497970ab9bbce4a8c6e9896b0f4e96f5269a0",
"triggerType" : "PUSH"
}, {
"hash" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2400",
"triggerID" : "b2d8b3f67e3556ccb84309b381ea513731537f8b",
"triggerType" : "PUSH"
}, {
"hash" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2420",
"triggerID" : "b5e0b228840b7c0575b6eaaf888e0f5bb6024b91",
"triggerType" : "PUSH"
}, {
"hash" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2422",
"triggerID" : "c0f0a8e1025051d6077e168b4ca15f4ee8edccc1",
"triggerType" : "PUSH"
}, {
"hash" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2459",
"triggerID" : "42ca73a97f00023736feeda07e96d03e7a621c9b",
"triggerType" : "PUSH"
}, {
"hash" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2460",
"triggerID" : "0fe30385aaabcc0b7f1474228d397ea67e174804",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2461",
"triggerID" : "a423a63a9a530365b531a4199a5010dd52708d86",
"triggerType" : "PUSH"
}, {
"hash" : "a423a63a9a530365b531a4199a5010dd52708d86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2466",
"triggerID" : "930677502",
"triggerType" : "MANUAL"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2566",
"triggerID" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"triggerType" : "PUSH"
}, {
"hash" : "4af82a25703eca83ff1e6e8d12055bcfec8357e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2574",
"triggerID" : "939599746",
"triggerType" : "MANUAL"
}, {
"hash" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576",
"triggerID" : "cdcf299c9f4e743cd74d9266603a30bf3301c3b8",
"triggerType" : "PUSH"
}, {
"hash" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619",
"triggerID" : "89dab78876b2512aa4967ced70da27f6fdb46b14",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cdcf299c9f4e743cd74d9266603a30bf3301c3b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2576)
* 89dab78876b2512aa4967ced70da27f6fdb46b14 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2619)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r711561579
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
+ if (datum == null) {
+ return null;
+ }
+ try {
+ TokenBuffer generator = new TokenBuffer(new ObjectMapper(), false);
+ toJson(datum, generator);
+ return new ObjectMapper().readTree(generator.asParser());
+ } catch (IOException e) {
+ throw new HoodieSchemaException(e);
+ }
+ }
+
+ private static void toJson(Object datum, JsonGenerator generator) throws IOException {
+ if (datum == JsonProperties.NULL_VALUE) { // null
+ generator.writeNull();
+ } else if (datum instanceof Map) { // record, map
+ generator.writeStartObject();
+ for (Map.Entry<Object, Object> entry : ((Map<Object, Object>) datum).entrySet()) {
+ generator.writeFieldName(entry.getKey().toString());
+ toJson(entry.getValue(), generator);
+ }
+ generator.writeEndObject();
+ } else if (datum instanceof Collection) { // array
+ generator.writeStartArray();
+ for (Object element : (Collection) datum) {
+ toJson(element, generator);
+ }
+ generator.writeEndArray();
+ } else if (datum instanceof byte[]) { // bytes, fixed
+ generator.writeString(new String((byte[]) datum, StandardCharsets.ISO_8859_1));
+ } else if (datum instanceof CharSequence || datum instanceof Enum) { // string, enum
+ generator.writeString(datum.toString());
+ } else if (datum instanceof Double) { // double
+ generator.writeNumber((Double) datum);
+ } else if (datum instanceof Float) { // float
+ generator.writeNumber((Float) datum);
+ } else if (datum instanceof Long) { // long
+ generator.writeNumber((Long) datum);
+ } else if (datum instanceof Integer) { // int
+ generator.writeNumber((Integer) datum);
+ } else if (datum instanceof Boolean) { // boolean
+ generator.writeBoolean((Boolean) datum);
+ } else {
+ throw new HoodieSchemaException("Unknown datum class: " + datum.getClass());
+ }
+ }
+
+ public boolean isValidDefaultValue(Type type, JsonNode defaultValue) {
+ if (defaultValue == null) {
+ return true;
+ }
+ switch (type.typeId()) {
+ case STRING:
+ case BINARY:
+ case FIXED:
+ return defaultValue.isTextual();
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ return defaultValue.isNumber();
+ case BOOLEAN:
+ return defaultValue.isBoolean();
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType)type;
+ if (!defaultValue.isArray()) {
+ return false;
+ }
+ for (JsonNode element : defaultValue) {
+ if (!isValidDefaultValue(array.elementType(), element)) {
+ return false;
+ }
+ }
+ return true;
+ case MAP:
+ Types.MapType map = (Types.MapType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (JsonNode value : defaultValue) {
+ if (!isValidDefaultValue(map.valueType(), value)) {
+ return false;
+ }
+ }
+ return true;
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (Types.Field f : record.fields()) {
+ if (!isValidDefaultValue(f.type(), defaultValue.has(f.name()) ? defaultValue.get(f.name()) : toJsonNode(f.getDefaultValue()))) {
+ return false;
+ }
+ }
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ public static Object toObject(JsonNode jsonNode, Type type) {
+ if (jsonNode == null) {
+ return null;
+ } else if (jsonNode.isNull()) {
+ return null;
+ } else if (jsonNode.isBoolean()) {
+ return jsonNode.asBoolean();
+ } else if (jsonNode.isInt()) {
+ if (type == null || type.typeId().equals(Type.TypeID.INT)) {
+ return jsonNode.asInt();
+ } else if (type.typeId().equals(Type.TypeID.LONG)) {
+ return jsonNode.asLong();
+ }
+ } else if (jsonNode.isLong()) {
+ return jsonNode.asLong();
+ } else if (jsonNode.isDouble() || jsonNode.isFloat()) {
+ if (type == null || type.typeId().equals(Type.TypeID.DOUBLE)) {
+ return jsonNode.asDouble();
+ } else if (type.typeId().equals(Type.TypeID.FLOAT)) {
+ return (float) jsonNode.asDouble();
+ }
+ } else if (jsonNode.isTextual()) {
+ if (type == null || type.typeId().equals(Type.TypeID.STRING)) {
+ return jsonNode.asText();
+ } else if (type.typeId().equals(Type.TypeID.BINARY) || type.typeId().equals(Type.TypeID.FIXED)) {
+ return jsonNode.textValue().getBytes(StandardCharsets.ISO_8859_1);
+ }
+ } else if (jsonNode.isArray()) {
+ List<Object> l = new ArrayList<>();
+ for (JsonNode node : jsonNode) {
+ l.add(toObject(node, type == null ? null : ((Types.ArrayType)type).elementType()));
+ }
+ return l;
+ } else if (jsonNode.isObject()) {
+ Map<Object, Object> m = new LinkedHashMap<>();
+ for (Iterator<String> it = jsonNode.fieldNames(); it.hasNext();) {
+ String key = it.next();
+ final Type s;
+ if (type != null && type.typeId().equals(Type.TypeID.MAP)) {
+ s = ((Types.MapType)type).valueType();
+ } else if (type != null && type.typeId().equals(Type.TypeID.RECORD)) {
+ s = ((Types.RecordType)type).field(key).type();
+ } else {
+ s = null;
+ }
+ Object value = toObject(jsonNode.get(key), s);
+ m.put(key, value);
+ }
+ return m;
+ }
+ return null;
+ }
+
+ public static String toJson(List<InternalSchema> internalSchemas) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ generator.writeStartObject();
+ generator.writeArrayFieldStart(SCHEMAS);
+ for (InternalSchema schema : internalSchemas) {
+ toJson(schema, generator);
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static String toJson(InternalSchema internalSchema) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ toJson(internalSchema, generator);
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void toJson(InternalSchema internalSchema, JsonGenerator generator) throws IOException {
+ toJson(internalSchema.getRecord(), internalSchema.getMax_column_id(), internalSchema.schemaId(), generator);
+ }
+
+ public static void toJson(Types.RecordType record, Integer maxColumnId, Long versionId, JsonGenerator generator) throws IOException {
+ generator.writeStartObject();
+ if (maxColumnId != null) {
+ generator.writeNumberField(MAX_COLUMN_ID, maxColumnId);
+ }
+ if (versionId != null) {
+ generator.writeNumberField(VERSION_ID, versionId);
+ }
+ generator.writeStringField(TYPE, RECORD);
+ generator.writeArrayFieldStart(FIELDS);
+ for (Types.Field field : record.fields()) {
+ generator.writeStartObject();
+ generator.writeNumberField(ID, field.fieldId());
+ generator.writeStringField(NAME, field.name());
+ generator.writeBooleanField(OPTIONAL, field.isOptional());
+ generator.writeFieldName(TYPE);
+ toJson(field.type(), generator);
+ if (field.doc() != null) {
+ generator.writeStringField(DOC, field.doc());
+ }
+ generator.writeEndObject();
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ }
+
+ static void toJson(Type type, JsonGenerator generator) throws IOException {
+ switch (type.typeId()) {
+ case RECORD:
+ toJson((Types.RecordType) type, null, null, generator);
+ break;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, ARRAY);
+ generator.writeNumberField(ELEMENT_ID, array.elementId());
+ generator.writeFieldName(ELEMENT);
+ toJson(array.elementType(), generator);
+ generator.writeBooleanField(ELEMENT_OPTIONAL, array.isElementOptional());
+ generator.writeEndObject();
+ break;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, MAP);
+ generator.writeNumberField(KEY_ID, map.keyId());
+ generator.writeFieldName(KEY);
+ toJson(map.keyType(), generator);
+ generator.writeNumberField(VALUE_ID, map.valueId());
+ generator.writeFieldName(VALUE);
+ toJson(map.valueType(), generator);
+ generator.writeBooleanField(VALUE_OPTIONAL, map.isValueOptional());
+ generator.writeEndObject();
+ break;
+ default:
+ if (!type.isNestedType()) {
+ generator.writeString(type.toString());
+ } else {
+ throw new IllegalArgumentIOException(String.format("cannot write unknown types: %s", type));
+ }
+ }
+ }
+
+ private static Type parserTypeFromJson(JsonNode jsonNode) {
+ if (jsonNode.isTextual()) {
+ String type = jsonNode.asText().toLowerCase(Locale.ROOT);
+ // deal with fixed and decimal
+ Matcher fixed = FIXED.matcher(type);
+ if (fixed.matches()) {
+ return Types.FixedType.getFixed(Integer.parseInt(fixed.group(1)));
+ }
+ Matcher decimal = DECIMAL.matcher(type);
+ if (decimal.matches()) {
+ return Types.DecimalType.get(
+ Integer.parseInt(decimal.group(1)),
+ Integer.parseInt(decimal.group(2)));
+ }
+ // deal with other type
+ switch (Type.fromValue(type)) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIME:
+ return Types.TimeType.get();
+ case TIMESTAMP:
+ return Types.TimestampType.get();
+ case STRING:
+ return Types.StringType.get();
+ case UUID:
+ return Types.UUIDType.get();
+ case BINARY:
+ return Types.BinaryType.get();
+ default:
+ throw new IllegalArgumentException("cannot parser types from jsonNode");
+ }
+ } else if (jsonNode.isObject()) {
+ String typeStr = jsonNode.get(TYPE).asText();
+ if (RECORD.equals(typeStr)) {
+ JsonNode fieldNodes = jsonNode.get(FIELDS);
+ Iterator<JsonNode> iter = fieldNodes.elements();
+ List<Types.Field> fields = new ArrayList<>();
+ while (iter.hasNext()) {
+ JsonNode field = iter.next();
+ // extract
+ int id = field.get(ID).asInt();
+ String name = field.get(NAME).asText();
+ Type type = parserTypeFromJson(field.get(TYPE));
+ String doc = field.has(DOC) ? field.get(DOC).asText() : null;
+ boolean optional = field.get(OPTIONAL).asBoolean();
+ // build fields
+ fields.add(Types.Field.get(id, optional, name, type, doc));
+ }
+ return Types.RecordType.get(fields);
+ } else if (ARRAY.equals(typeStr)) {
+ int elementId = jsonNode.get(ELEMENT_ID).asInt();
+ Type elementType = parserTypeFromJson(jsonNode.get(ELEMENT));
+ boolean optional = jsonNode.get(ELEMENT_OPTIONAL).asBoolean();
+ return Types.ArrayType.get(elementId, optional, elementType);
+ } else if (MAP.equals(typeStr)) {
+ int keyId = jsonNode.get(KEY_ID).asInt();
+ Type keyType = parserTypeFromJson(jsonNode.get(KEY));
+ int valueId = jsonNode.get(VALUE_ID).asInt();
+ Type valueType = parserTypeFromJson(jsonNode.get(VALUE));
+ boolean optional = jsonNode.get(VALUE_OPTIONAL).asBoolean();
+ return Types.MapType.get(keyId, valueId, keyType, valueType, optional);
+ }
+ }
+ throw new IllegalArgumentException(String.format("cannot parse type from jsonNode: %s", jsonNode));
+ }
+
+ public static InternalSchema fromJson(JsonNode jsonNode) {
+ Integer maxColumnId = !jsonNode.has(MAX_COLUMN_ID) ? null : jsonNode.get(MAX_COLUMN_ID).asInt();
+ Long versionId = !jsonNode.has(VERSION_ID) ? null : jsonNode.get(VERSION_ID).asLong();
+ Types.RecordType type = (Types.RecordType)parserTypeFromJson(jsonNode);
+ if (versionId == null) {
+ return new InternalSchema(type.fields());
+ } else {
+ if (maxColumnId != null) {
+ return new InternalSchema(versionId, maxColumnId, type.fields());
+ } else {
+ return new InternalSchema(versionId, type.fields());
+ }
+ }
+ }
+
+ public static InternalSchema fromJson(String json) {
+ if (json == null || json.isEmpty()) {
+ return null;
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709163345
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -1068,4 +1076,150 @@ public void close() {
this.heartbeatClient.stop();
this.txnManager.close();
}
+
+ /**
+ * add columns to table.
+ *
+ * @param colName col name to be added. if we want to add col to a nested filed, the fullName should be specify
+ * @param schema col type to be added.
+ * @param doc col doc to be added.
+ * @param position col position to be added
+ * @param positionType col position change type. now support three change types: first/after/before
+ */
+ public void addColumns(String colName, Schema schema, String doc, String position, String positionType) {
+ Pair<InternalSchema, HoodieTableMetaClient> pair = getInternalSchemaAndMetaClient();
+ TableChanges.ColumnAddChange add = TableChanges.ColumnAddChange.get(pair.getLeft());
+ String parentName = TableChanges.getParentName(colName);
+ add.addColumns(parentName, colName, AvroSchemaUtil.convertToField(schema), doc);
+ if (position != null && !position.isEmpty() && positionType != null && !position.isEmpty()) {
Review comment:
the last `!position.isEmpty()` change to `!positionType.isEmpty()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717297644
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,779 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ /**
+ * build a mapping from id to full field name for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to full field name
+ */
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ /**
+ * build a mapping from full field name to id for a internal Type.
+ * if a field y belong to a struct filed x, then the full name of y is x.y
+ *
+ * @param type hoodie internal type
+ * @return a mapping from full field name to id
+ */
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ /**
+ * build a mapping from id to field for a internal Type.
+ *
+ * @param type hoodie internal type
+ * @return a mapping from id to field
+ */
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r717296982
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,542 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
Review comment:
done, docs added, and rename InternalSchemaParser to SerDeHelper
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710696033
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+/**
+ * helper class.
+ * use static cache to cache historySchema info to prevent fetching from hdfs every time.
+ * all task belong to the same executor/container will share this cache.
+ *
+ */
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
+ HISTORICAL_SCHEMA_CACHE = Caffeine.newBuilder().maximumSize(1000).weakValues().build();
+
+ public static InternalSchema searchSchemaAndCache(long versionID, String tablePath, Configuration hadoopConf) {
+ TreeMap<Long, InternalSchema> historicalSchemas = HISTORICAL_SCHEMA_CACHE.getIfPresent(tablePath);
+ if (historicalSchemas == null || InternalSchemaParser.searchSchema(versionID, historicalSchemas) == null) {
+ historicalSchemas = getHistoricalSchemas(tablePath, hadoopConf);
+ HISTORICAL_SCHEMA_CACHE.put(tablePath, historicalSchemas);
+ } else {
+ long maxVersionId = historicalSchemas.keySet().stream().max(Long::compareTo).get();
Review comment:
Let's use lastKey() instead of streaming and comparing since it's a TreeMap
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Internal schema for hudi table.
+ * used to support schema evolution.
+ */
+public class InternalSchema implements Serializable {
+
+ private static final long DEFAULT_VERSION_ID = 0;
+
+ private final RecordType record;
+
+ private int maxColumnId;
+ private long versionId;
+
+ private transient Map<Integer, Field> idToField = null;
+ private transient Map<String, Integer> nameToId = null;
+ private transient Map<Integer, String> idToName = null;
+
+ public InternalSchema(List columns) {
+ this(DEFAULT_VERSION_ID, columns);
+ }
+
+ public InternalSchema(Field... columns) {
+ this(DEFAULT_VERSION_ID, Arrays.asList(columns));
+ }
+
+ public InternalSchema(long versionId, List<Field> cols) {
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ maxColumnId = idToName.keySet().stream().max(Comparator.comparing(Integer::valueOf)).get();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, List<Field> cols) {
+ this.maxColumnId = maxColumnId;
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, Field... cols) {
+ this(versionId, maxColumnId, Arrays.asList(cols));
+ }
+
+ public RecordType getRecord() {
+ return record;
+ }
+
+ private Map<Integer, String> buildIdToName() {
+ if (idToName == null) {
+ idToName = InternalSchemaUtils.buildIdToName(record);
+ }
+ return idToName;
+ }
+
+ private Map<String, Integer> buildNameToId() {
+ if (nameToId == null) {
+ nameToId = InternalSchemaUtils.buildNameToId(record);
Review comment:
Before calling we can also check if `idToName` is not null or empty then return inverse map. We can save the recursive calls in `InternalSchemaUtils.buildNameToId(record)`.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/TableInternalSchemaUtils.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.utils.InternalSchemaParser;
+
+import java.util.TreeMap;
+
+/**
+ * helper class.
+ * use static cache to cache historySchema info to prevent fetching from hdfs every time.
+ * all task belong to the same executor/container will share this cache.
+ *
+ */
+public class TableInternalSchemaUtils {
+ private static final Cache<String, TreeMap<Long, InternalSchema>>
+ HISTORICAL_SCHEMA_CACHE = Caffeine.newBuilder().maximumSize(1000).weakValues().build();
+
+ public static InternalSchema searchSchemaAndCache(long versionID, String tablePath, Configuration hadoopConf) {
+ TreeMap<Long, InternalSchema> historicalSchemas = HISTORICAL_SCHEMA_CACHE.getIfPresent(tablePath);
+ if (historicalSchemas == null || InternalSchemaParser.searchSchema(versionID, historicalSchemas) == null) {
+ historicalSchemas = getHistoricalSchemas(tablePath, hadoopConf);
+ HISTORICAL_SCHEMA_CACHE.put(tablePath, historicalSchemas);
+ } else {
+ long maxVersionId = historicalSchemas.keySet().stream().max(Long::compareTo).get();
+ if (versionID > maxVersionId) {
+ historicalSchemas = getHistoricalSchemas(tablePath, hadoopConf);
+ HISTORICAL_SCHEMA_CACHE.put(tablePath, historicalSchemas);
+ }
+ }
+ return InternalSchemaParser.searchSchema(versionID, historicalSchemas);
+ }
+
+ private static TreeMap<Long, InternalSchema> getHistoricalSchemas(String tablePath, Configuration hadoopConf) {
+ TreeMap<Long, InternalSchema> result = new TreeMap<>();
Review comment:
What about thread-safety? If there are concurrent readers, then should construction of TreeMap be synchronized?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
+ if (datum == null) {
+ return null;
+ }
+ try {
+ TokenBuffer generator = new TokenBuffer(new ObjectMapper(), false);
+ toJson(datum, generator);
+ return new ObjectMapper().readTree(generator.asParser());
+ } catch (IOException e) {
+ throw new HoodieSchemaException(e);
+ }
+ }
+
+ private static void toJson(Object datum, JsonGenerator generator) throws IOException {
+ if (datum == JsonProperties.NULL_VALUE) { // null
+ generator.writeNull();
+ } else if (datum instanceof Map) { // record, map
+ generator.writeStartObject();
+ for (Map.Entry<Object, Object> entry : ((Map<Object, Object>) datum).entrySet()) {
+ generator.writeFieldName(entry.getKey().toString());
+ toJson(entry.getValue(), generator);
+ }
+ generator.writeEndObject();
+ } else if (datum instanceof Collection) { // array
+ generator.writeStartArray();
+ for (Object element : (Collection) datum) {
+ toJson(element, generator);
+ }
+ generator.writeEndArray();
+ } else if (datum instanceof byte[]) { // bytes, fixed
+ generator.writeString(new String((byte[]) datum, StandardCharsets.ISO_8859_1));
+ } else if (datum instanceof CharSequence || datum instanceof Enum) { // string, enum
+ generator.writeString(datum.toString());
+ } else if (datum instanceof Double) { // double
+ generator.writeNumber((Double) datum);
+ } else if (datum instanceof Float) { // float
+ generator.writeNumber((Float) datum);
+ } else if (datum instanceof Long) { // long
+ generator.writeNumber((Long) datum);
+ } else if (datum instanceof Integer) { // int
+ generator.writeNumber((Integer) datum);
+ } else if (datum instanceof Boolean) { // boolean
+ generator.writeBoolean((Boolean) datum);
+ } else {
+ throw new HoodieSchemaException("Unknown datum class: " + datum.getClass());
+ }
+ }
+
+ public boolean isValidDefaultValue(Type type, JsonNode defaultValue) {
+ if (defaultValue == null) {
+ return true;
+ }
+ switch (type.typeId()) {
+ case STRING:
+ case BINARY:
+ case FIXED:
+ return defaultValue.isTextual();
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ return defaultValue.isNumber();
+ case BOOLEAN:
+ return defaultValue.isBoolean();
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType)type;
+ if (!defaultValue.isArray()) {
+ return false;
+ }
+ for (JsonNode element : defaultValue) {
+ if (!isValidDefaultValue(array.elementType(), element)) {
+ return false;
+ }
+ }
+ return true;
+ case MAP:
+ Types.MapType map = (Types.MapType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (JsonNode value : defaultValue) {
+ if (!isValidDefaultValue(map.valueType(), value)) {
+ return false;
+ }
+ }
+ return true;
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (Types.Field f : record.fields()) {
+ if (!isValidDefaultValue(f.type(), defaultValue.has(f.name()) ? defaultValue.get(f.name()) : toJsonNode(f.getDefaultValue()))) {
+ return false;
+ }
+ }
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ public static Object toObject(JsonNode jsonNode, Type type) {
+ if (jsonNode == null) {
+ return null;
+ } else if (jsonNode.isNull()) {
+ return null;
+ } else if (jsonNode.isBoolean()) {
+ return jsonNode.asBoolean();
+ } else if (jsonNode.isInt()) {
+ if (type == null || type.typeId().equals(Type.TypeID.INT)) {
+ return jsonNode.asInt();
+ } else if (type.typeId().equals(Type.TypeID.LONG)) {
+ return jsonNode.asLong();
+ }
+ } else if (jsonNode.isLong()) {
+ return jsonNode.asLong();
+ } else if (jsonNode.isDouble() || jsonNode.isFloat()) {
+ if (type == null || type.typeId().equals(Type.TypeID.DOUBLE)) {
+ return jsonNode.asDouble();
+ } else if (type.typeId().equals(Type.TypeID.FLOAT)) {
+ return (float) jsonNode.asDouble();
+ }
+ } else if (jsonNode.isTextual()) {
+ if (type == null || type.typeId().equals(Type.TypeID.STRING)) {
+ return jsonNode.asText();
+ } else if (type.typeId().equals(Type.TypeID.BINARY) || type.typeId().equals(Type.TypeID.FIXED)) {
+ return jsonNode.textValue().getBytes(StandardCharsets.ISO_8859_1);
+ }
+ } else if (jsonNode.isArray()) {
+ List<Object> l = new ArrayList<>();
+ for (JsonNode node : jsonNode) {
+ l.add(toObject(node, type == null ? null : ((Types.ArrayType)type).elementType()));
+ }
+ return l;
+ } else if (jsonNode.isObject()) {
+ Map<Object, Object> m = new LinkedHashMap<>();
+ for (Iterator<String> it = jsonNode.fieldNames(); it.hasNext();) {
+ String key = it.next();
+ final Type s;
+ if (type != null && type.typeId().equals(Type.TypeID.MAP)) {
+ s = ((Types.MapType)type).valueType();
+ } else if (type != null && type.typeId().equals(Type.TypeID.RECORD)) {
+ s = ((Types.RecordType)type).field(key).type();
+ } else {
+ s = null;
+ }
+ Object value = toObject(jsonNode.get(key), s);
+ m.put(key, value);
+ }
+ return m;
+ }
+ return null;
+ }
+
+ public static String toJson(List<InternalSchema> internalSchemas) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ generator.writeStartObject();
+ generator.writeArrayFieldStart(SCHEMAS);
+ for (InternalSchema schema : internalSchemas) {
+ toJson(schema, generator);
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static String toJson(InternalSchema internalSchema) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ toJson(internalSchema, generator);
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void toJson(InternalSchema internalSchema, JsonGenerator generator) throws IOException {
+ toJson(internalSchema.getRecord(), internalSchema.getMax_column_id(), internalSchema.schemaId(), generator);
+ }
+
+ public static void toJson(Types.RecordType record, Integer maxColumnId, Long versionId, JsonGenerator generator) throws IOException {
+ generator.writeStartObject();
+ if (maxColumnId != null) {
+ generator.writeNumberField(MAX_COLUMN_ID, maxColumnId);
+ }
+ if (versionId != null) {
+ generator.writeNumberField(VERSION_ID, versionId);
+ }
+ generator.writeStringField(TYPE, RECORD);
+ generator.writeArrayFieldStart(FIELDS);
+ for (Types.Field field : record.fields()) {
+ generator.writeStartObject();
+ generator.writeNumberField(ID, field.fieldId());
+ generator.writeStringField(NAME, field.name());
+ generator.writeBooleanField(OPTIONAL, field.isOptional());
+ generator.writeFieldName(TYPE);
+ toJson(field.type(), generator);
+ if (field.doc() != null) {
+ generator.writeStringField(DOC, field.doc());
+ }
+ generator.writeEndObject();
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ }
+
+ static void toJson(Type type, JsonGenerator generator) throws IOException {
+ switch (type.typeId()) {
+ case RECORD:
+ toJson((Types.RecordType) type, null, null, generator);
+ break;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, ARRAY);
+ generator.writeNumberField(ELEMENT_ID, array.elementId());
+ generator.writeFieldName(ELEMENT);
+ toJson(array.elementType(), generator);
+ generator.writeBooleanField(ELEMENT_OPTIONAL, array.isElementOptional());
+ generator.writeEndObject();
+ break;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, MAP);
+ generator.writeNumberField(KEY_ID, map.keyId());
+ generator.writeFieldName(KEY);
+ toJson(map.keyType(), generator);
+ generator.writeNumberField(VALUE_ID, map.valueId());
+ generator.writeFieldName(VALUE);
+ toJson(map.valueType(), generator);
+ generator.writeBooleanField(VALUE_OPTIONAL, map.isValueOptional());
+ generator.writeEndObject();
+ break;
+ default:
+ if (!type.isNestedType()) {
+ generator.writeString(type.toString());
+ } else {
+ throw new IllegalArgumentIOException(String.format("cannot write unknown types: %s", type));
+ }
+ }
+ }
+
+ private static Type parserTypeFromJson(JsonNode jsonNode) {
+ if (jsonNode.isTextual()) {
+ String type = jsonNode.asText().toLowerCase(Locale.ROOT);
+ // deal with fixed and decimal
+ Matcher fixed = FIXED.matcher(type);
+ if (fixed.matches()) {
+ return Types.FixedType.getFixed(Integer.parseInt(fixed.group(1)));
+ }
+ Matcher decimal = DECIMAL.matcher(type);
+ if (decimal.matches()) {
+ return Types.DecimalType.get(
+ Integer.parseInt(decimal.group(1)),
+ Integer.parseInt(decimal.group(2)));
+ }
+ // deal with other type
+ switch (Type.fromValue(type)) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIME:
+ return Types.TimeType.get();
+ case TIMESTAMP:
+ return Types.TimestampType.get();
+ case STRING:
+ return Types.StringType.get();
+ case UUID:
+ return Types.UUIDType.get();
+ case BINARY:
+ return Types.BinaryType.get();
+ default:
+ throw new IllegalArgumentException("cannot parser types from jsonNode");
+ }
+ } else if (jsonNode.isObject()) {
+ String typeStr = jsonNode.get(TYPE).asText();
+ if (RECORD.equals(typeStr)) {
+ JsonNode fieldNodes = jsonNode.get(FIELDS);
+ Iterator<JsonNode> iter = fieldNodes.elements();
+ List<Types.Field> fields = new ArrayList<>();
+ while (iter.hasNext()) {
+ JsonNode field = iter.next();
+ // extract
+ int id = field.get(ID).asInt();
+ String name = field.get(NAME).asText();
+ Type type = parserTypeFromJson(field.get(TYPE));
+ String doc = field.has(DOC) ? field.get(DOC).asText() : null;
+ boolean optional = field.get(OPTIONAL).asBoolean();
+ // build fields
+ fields.add(Types.Field.get(id, optional, name, type, doc));
+ }
+ return Types.RecordType.get(fields);
+ } else if (ARRAY.equals(typeStr)) {
+ int elementId = jsonNode.get(ELEMENT_ID).asInt();
+ Type elementType = parserTypeFromJson(jsonNode.get(ELEMENT));
+ boolean optional = jsonNode.get(ELEMENT_OPTIONAL).asBoolean();
+ return Types.ArrayType.get(elementId, optional, elementType);
+ } else if (MAP.equals(typeStr)) {
+ int keyId = jsonNode.get(KEY_ID).asInt();
+ Type keyType = parserTypeFromJson(jsonNode.get(KEY));
+ int valueId = jsonNode.get(VALUE_ID).asInt();
+ Type valueType = parserTypeFromJson(jsonNode.get(VALUE));
+ boolean optional = jsonNode.get(VALUE_OPTIONAL).asBoolean();
+ return Types.MapType.get(keyId, valueId, keyType, valueType, optional);
+ }
+ }
+ throw new IllegalArgumentException(String.format("cannot parse type from jsonNode: %s", jsonNode));
+ }
+
+ public static InternalSchema fromJson(JsonNode jsonNode) {
+ Integer maxColumnId = !jsonNode.has(MAX_COLUMN_ID) ? null : jsonNode.get(MAX_COLUMN_ID).asInt();
+ Long versionId = !jsonNode.has(VERSION_ID) ? null : jsonNode.get(VERSION_ID).asLong();
+ Types.RecordType type = (Types.RecordType)parserTypeFromJson(jsonNode);
+ if (versionId == null) {
+ return new InternalSchema(type.fields());
+ } else {
+ if (maxColumnId != null) {
+ return new InternalSchema(versionId, maxColumnId, type.fields());
+ } else {
+ return new InternalSchema(versionId, type.fields());
+ }
+ }
+ }
+
+ public static InternalSchema fromJson(String json) {
+ if (json == null || json.isEmpty()) {
+ return null;
Review comment:
Shall we return `Optional<InternalSchema>` so that it becomes safer to handle at the caller site?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
Review comment:
Let's add docs for public methods, and `requireNonNull` for the parameters if necessary.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Internal schema for hudi table.
+ * used to support schema evolution.
+ */
+public class InternalSchema implements Serializable {
+
+ private static final long DEFAULT_VERSION_ID = 0;
+
+ private final RecordType record;
+
+ private int maxColumnId;
+ private long versionId;
+
+ private transient Map<Integer, Field> idToField = null;
+ private transient Map<String, Integer> nameToId = null;
+ private transient Map<Integer, String> idToName = null;
+
+ public InternalSchema(List columns) {
+ this(DEFAULT_VERSION_ID, columns);
+ }
+
+ public InternalSchema(Field... columns) {
+ this(DEFAULT_VERSION_ID, Arrays.asList(columns));
+ }
+
+ public InternalSchema(long versionId, List<Field> cols) {
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ maxColumnId = idToName.keySet().stream().max(Comparator.comparing(Integer::valueOf)).get();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, List<Field> cols) {
+ this.maxColumnId = maxColumnId;
+ this.versionId = versionId;
+ this.record = RecordType.get(cols);
+ buildIdToName();
+ }
+
+ public InternalSchema(long versionId, int maxColumnId, Field... cols) {
+ this(versionId, maxColumnId, Arrays.asList(cols));
+ }
+
+ public RecordType getRecord() {
+ return record;
+ }
+
+ private Map<Integer, String> buildIdToName() {
+ if (idToName == null) {
+ idToName = InternalSchemaUtils.buildIdToName(record);
+ }
+ return idToName;
+ }
+
+ private Map<String, Integer> buildNameToId() {
+ if (nameToId == null) {
+ nameToId = InternalSchemaUtils.buildNameToId(record);
+ }
+ return nameToId;
+ }
+
+ private Map<Integer, Field> buildIdToField() {
+ if (idToField == null) {
+ idToField = InternalSchemaUtils.buildIdToField(record);
+ }
+ return idToField;
+ }
+
+ /**
+ * get all columns full name.
+ */
+ public List<String> getAllColsFullName() {
+ if (nameToId == null) {
+ nameToId = InternalSchemaUtils.buildNameToId(record);
+ }
+ return Arrays.asList(nameToId.keySet().toArray(new String[0]));
+ }
+
+ /**
+ * set the version ID for this schema.
+ */
+ public InternalSchema setSchemaId(long versionId) {
+ this.versionId = versionId;
+ return this;
+ }
+
+ /**
+ * Returns the version ID for this schema.
+ */
+ public long schemaId() {
+ return this.versionId;
+ }
+
+ /**
+ * set the version ID for this schema.
+ */
+ public void setMax_column_id(int maxColumnId) {
+ this.maxColumnId = maxColumnId;
+ }
+
+ /**
+ * Returns the max column id for this schema.
+ */
+ public int getMax_column_id() {
+ return this.maxColumnId;
+ }
+
+ /**
+ * Returns a List of the {@link Field columns} in this Schema.
+ */
+ public List<Field> columns() {
+ return record.fields();
+ }
+
+ /**
+ * Returns the {@link Type} of a sub-field identified by the field name.
+ * @param id a field id
+ * @return fullName of field of
+ */
+ public String findfullName(int id) {
+ if (idToName == null) {
+ buildIdToName();
+ }
+ return idToName.get(id);
+ }
+
+ /**
+ * Returns the {@link Type} of a sub-field identified by the field name.
+ * @param name a field name
+ * @return a Type for the sub-field or null if it is not found
+ */
+ public Type findType(String name) {
+ if (name == null || name.isEmpty()) {
+ return null;
+ }
+ Integer id = buildNameToId().get(name);
+ if (id != null) { // name is found
+ return findType(id);
+ }
+ return null;
+ }
+
+ /**
+ * Returns the {@link Type} of a sub-field identified by the field id.
+ * @param id a field id
+ * @return a Type for the sub-field or null if it is not found
+ */
+ public Type findType(int id) {
+ Field field = buildIdToField().get(id);
+ if (field != null) {
+ return field.type();
+ }
+ return null;
+ }
+
+ /**
+ * Returns the sub-field identified by the field id.
+ * @param id a field id
+ * @return the sub-field or null if it is not found
+ */
+ public Field findField(int id) {
+ return buildIdToField().get(id);
+ }
+
+ /**
+ * Returns a sub-field by name as a {@link Field}.
+ * The result may be a top-level or a nested field.
+ * @param name a String name
+ * @return a Type for the sub-field or null if it is not found
+ */
+ public Field findField(String name) {
+ if (name == null || name.isEmpty()) {
+ return null;
+ }
+ Integer id = buildNameToId().get(name);
+ if (id != null) {
+ return buildIdToField().get(id);
+ }
+ return null;
+ }
+
+ public int findIdByName(String name) {
+ if (name == null || name.isEmpty()) {
Review comment:
Let's also do `!buildNameToId().containsKey(name)` check here, or `buildNameToId().getOrDefault(name, -1)` on the return line.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/AvroSchemaUtil.java
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import static org.apache.avro.Schema.Type.UNION;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.action.SchemaChangeUtils;
+import org.apache.hudi.internal.schema.action.TableChanges;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+public class AvroSchemaUtil {
+ private AvroSchemaUtil() {
+ }
+
+ public static Schema convert(InternalSchema internalSchema, String tableName) {
+ return InternalSchemaUtils.buildAvroSchemaFromInternalSchema(internalSchema, tableName);
+ }
+
+ public static Schema convert(Types.RecordType type, String name) {
+ return InternalSchemaUtils.buildAvroSchemaFromType(type, name);
+ }
+
+ public static Schema convert(Type type, String name) {
+ return InternalSchemaUtils.buildAvroSchemaFromType(type, name);
+ }
+
+ public static Type convertToField(Schema schema) {
+ return InternalSchemaUtils.buildTypeFromAvroSchema(schema);
+ }
+
+ public static InternalSchema convert(Schema schema) {
+ List<Types.Field> fields = ((Types.RecordType) convertToField(schema)).fields();
+ return new InternalSchema(fields);
+ }
+
+ public static boolean isOptional(Schema schema) {
+ if (schema.getType() == UNION && schema.getTypes().size() == 2) {
+ if (schema.getTypes().get(0).getType() == Schema.Type.NULL) {
Review comment:
This can be simplified to:
```
if (schema.getType() == UNION && schema.getTypes().size() == 2) {
return schema.getTypes().get(0).getType() == Schema.Type.NULL || schema.getTypes().get(1).getType() == Schema.Type.NULL;
}
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/InternalSchema.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.utils.InternalSchemaUtils;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Internal schema for hudi table.
+ * used to support schema evolution.
+ */
+public class InternalSchema implements Serializable {
+
+ private static final long DEFAULT_VERSION_ID = 0;
+
+ private final RecordType record;
+
+ private int maxColumnId;
+ private long versionId;
+
+ private transient Map<Integer, Field> idToField = null;
Review comment:
Just wondering if we should make only `idToField` non-transient and serialize this field? What do you think?
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/action/SchemaChangeUtils.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.action;
+
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Helper methods for schema Change.
+ */
+public class SchemaChangeUtils {
Review comment:
Let's move this to `internal.schema.utils` package.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
+ if (datum == null) {
+ return null;
+ }
+ try {
+ TokenBuffer generator = new TokenBuffer(new ObjectMapper(), false);
+ toJson(datum, generator);
+ return new ObjectMapper().readTree(generator.asParser());
+ } catch (IOException e) {
+ throw new HoodieSchemaException(e);
+ }
+ }
+
+ private static void toJson(Object datum, JsonGenerator generator) throws IOException {
+ if (datum == JsonProperties.NULL_VALUE) { // null
+ generator.writeNull();
+ } else if (datum instanceof Map) { // record, map
+ generator.writeStartObject();
+ for (Map.Entry<Object, Object> entry : ((Map<Object, Object>) datum).entrySet()) {
+ generator.writeFieldName(entry.getKey().toString());
+ toJson(entry.getValue(), generator);
+ }
+ generator.writeEndObject();
+ } else if (datum instanceof Collection) { // array
+ generator.writeStartArray();
+ for (Object element : (Collection) datum) {
+ toJson(element, generator);
+ }
+ generator.writeEndArray();
+ } else if (datum instanceof byte[]) { // bytes, fixed
+ generator.writeString(new String((byte[]) datum, StandardCharsets.ISO_8859_1));
+ } else if (datum instanceof CharSequence || datum instanceof Enum) { // string, enum
+ generator.writeString(datum.toString());
+ } else if (datum instanceof Double) { // double
+ generator.writeNumber((Double) datum);
+ } else if (datum instanceof Float) { // float
+ generator.writeNumber((Float) datum);
+ } else if (datum instanceof Long) { // long
+ generator.writeNumber((Long) datum);
+ } else if (datum instanceof Integer) { // int
+ generator.writeNumber((Integer) datum);
+ } else if (datum instanceof Boolean) { // boolean
+ generator.writeBoolean((Boolean) datum);
+ } else {
+ throw new HoodieSchemaException("Unknown datum class: " + datum.getClass());
+ }
+ }
+
+ public boolean isValidDefaultValue(Type type, JsonNode defaultValue) {
+ if (defaultValue == null) {
+ return true;
+ }
+ switch (type.typeId()) {
+ case STRING:
+ case BINARY:
+ case FIXED:
+ return defaultValue.isTextual();
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ return defaultValue.isNumber();
+ case BOOLEAN:
+ return defaultValue.isBoolean();
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType)type;
+ if (!defaultValue.isArray()) {
+ return false;
+ }
+ for (JsonNode element : defaultValue) {
+ if (!isValidDefaultValue(array.elementType(), element)) {
+ return false;
+ }
+ }
+ return true;
+ case MAP:
+ Types.MapType map = (Types.MapType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (JsonNode value : defaultValue) {
+ if (!isValidDefaultValue(map.valueType(), value)) {
+ return false;
+ }
+ }
+ return true;
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (Types.Field f : record.fields()) {
+ if (!isValidDefaultValue(f.type(), defaultValue.has(f.name()) ? defaultValue.get(f.name()) : toJsonNode(f.getDefaultValue()))) {
+ return false;
+ }
+ }
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ public static Object toObject(JsonNode jsonNode, Type type) {
+ if (jsonNode == null) {
+ return null;
+ } else if (jsonNode.isNull()) {
+ return null;
+ } else if (jsonNode.isBoolean()) {
+ return jsonNode.asBoolean();
+ } else if (jsonNode.isInt()) {
+ if (type == null || type.typeId().equals(Type.TypeID.INT)) {
+ return jsonNode.asInt();
+ } else if (type.typeId().equals(Type.TypeID.LONG)) {
+ return jsonNode.asLong();
+ }
+ } else if (jsonNode.isLong()) {
+ return jsonNode.asLong();
+ } else if (jsonNode.isDouble() || jsonNode.isFloat()) {
+ if (type == null || type.typeId().equals(Type.TypeID.DOUBLE)) {
+ return jsonNode.asDouble();
+ } else if (type.typeId().equals(Type.TypeID.FLOAT)) {
+ return (float) jsonNode.asDouble();
+ }
+ } else if (jsonNode.isTextual()) {
+ if (type == null || type.typeId().equals(Type.TypeID.STRING)) {
+ return jsonNode.asText();
+ } else if (type.typeId().equals(Type.TypeID.BINARY) || type.typeId().equals(Type.TypeID.FIXED)) {
+ return jsonNode.textValue().getBytes(StandardCharsets.ISO_8859_1);
+ }
+ } else if (jsonNode.isArray()) {
+ List<Object> l = new ArrayList<>();
+ for (JsonNode node : jsonNode) {
+ l.add(toObject(node, type == null ? null : ((Types.ArrayType)type).elementType()));
+ }
+ return l;
+ } else if (jsonNode.isObject()) {
+ Map<Object, Object> m = new LinkedHashMap<>();
+ for (Iterator<String> it = jsonNode.fieldNames(); it.hasNext();) {
+ String key = it.next();
+ final Type s;
+ if (type != null && type.typeId().equals(Type.TypeID.MAP)) {
+ s = ((Types.MapType)type).valueType();
+ } else if (type != null && type.typeId().equals(Type.TypeID.RECORD)) {
+ s = ((Types.RecordType)type).field(key).type();
+ } else {
+ s = null;
+ }
+ Object value = toObject(jsonNode.get(key), s);
+ m.put(key, value);
+ }
+ return m;
+ }
+ return null;
+ }
+
+ public static String toJson(List<InternalSchema> internalSchemas) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ generator.writeStartObject();
+ generator.writeArrayFieldStart(SCHEMAS);
+ for (InternalSchema schema : internalSchemas) {
+ toJson(schema, generator);
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static String toJson(InternalSchema internalSchema) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ toJson(internalSchema, generator);
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void toJson(InternalSchema internalSchema, JsonGenerator generator) throws IOException {
+ toJson(internalSchema.getRecord(), internalSchema.getMax_column_id(), internalSchema.schemaId(), generator);
+ }
+
+ public static void toJson(Types.RecordType record, Integer maxColumnId, Long versionId, JsonGenerator generator) throws IOException {
+ generator.writeStartObject();
+ if (maxColumnId != null) {
+ generator.writeNumberField(MAX_COLUMN_ID, maxColumnId);
+ }
+ if (versionId != null) {
+ generator.writeNumberField(VERSION_ID, versionId);
+ }
+ generator.writeStringField(TYPE, RECORD);
+ generator.writeArrayFieldStart(FIELDS);
+ for (Types.Field field : record.fields()) {
+ generator.writeStartObject();
+ generator.writeNumberField(ID, field.fieldId());
+ generator.writeStringField(NAME, field.name());
+ generator.writeBooleanField(OPTIONAL, field.isOptional());
+ generator.writeFieldName(TYPE);
+ toJson(field.type(), generator);
+ if (field.doc() != null) {
+ generator.writeStringField(DOC, field.doc());
+ }
+ generator.writeEndObject();
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ }
+
+ static void toJson(Type type, JsonGenerator generator) throws IOException {
+ switch (type.typeId()) {
+ case RECORD:
+ toJson((Types.RecordType) type, null, null, generator);
+ break;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, ARRAY);
+ generator.writeNumberField(ELEMENT_ID, array.elementId());
+ generator.writeFieldName(ELEMENT);
+ toJson(array.elementType(), generator);
+ generator.writeBooleanField(ELEMENT_OPTIONAL, array.isElementOptional());
+ generator.writeEndObject();
+ break;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, MAP);
+ generator.writeNumberField(KEY_ID, map.keyId());
+ generator.writeFieldName(KEY);
+ toJson(map.keyType(), generator);
+ generator.writeNumberField(VALUE_ID, map.valueId());
+ generator.writeFieldName(VALUE);
+ toJson(map.valueType(), generator);
+ generator.writeBooleanField(VALUE_OPTIONAL, map.isValueOptional());
+ generator.writeEndObject();
+ break;
+ default:
+ if (!type.isNestedType()) {
+ generator.writeString(type.toString());
+ } else {
+ throw new IllegalArgumentIOException(String.format("cannot write unknown types: %s", type));
+ }
+ }
+ }
+
+ private static Type parserTypeFromJson(JsonNode jsonNode) {
+ if (jsonNode.isTextual()) {
+ String type = jsonNode.asText().toLowerCase(Locale.ROOT);
+ // deal with fixed and decimal
+ Matcher fixed = FIXED.matcher(type);
+ if (fixed.matches()) {
+ return Types.FixedType.getFixed(Integer.parseInt(fixed.group(1)));
+ }
+ Matcher decimal = DECIMAL.matcher(type);
+ if (decimal.matches()) {
+ return Types.DecimalType.get(
+ Integer.parseInt(decimal.group(1)),
+ Integer.parseInt(decimal.group(2)));
+ }
+ // deal with other type
+ switch (Type.fromValue(type)) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIME:
+ return Types.TimeType.get();
+ case TIMESTAMP:
+ return Types.TimestampType.get();
+ case STRING:
+ return Types.StringType.get();
+ case UUID:
+ return Types.UUIDType.get();
+ case BINARY:
+ return Types.BinaryType.get();
+ default:
+ throw new IllegalArgumentException("cannot parser types from jsonNode");
+ }
+ } else if (jsonNode.isObject()) {
+ String typeStr = jsonNode.get(TYPE).asText();
+ if (RECORD.equals(typeStr)) {
+ JsonNode fieldNodes = jsonNode.get(FIELDS);
+ Iterator<JsonNode> iter = fieldNodes.elements();
+ List<Types.Field> fields = new ArrayList<>();
+ while (iter.hasNext()) {
+ JsonNode field = iter.next();
+ // extract
+ int id = field.get(ID).asInt();
+ String name = field.get(NAME).asText();
+ Type type = parserTypeFromJson(field.get(TYPE));
+ String doc = field.has(DOC) ? field.get(DOC).asText() : null;
+ boolean optional = field.get(OPTIONAL).asBoolean();
+ // build fields
+ fields.add(Types.Field.get(id, optional, name, type, doc));
+ }
+ return Types.RecordType.get(fields);
+ } else if (ARRAY.equals(typeStr)) {
+ int elementId = jsonNode.get(ELEMENT_ID).asInt();
+ Type elementType = parserTypeFromJson(jsonNode.get(ELEMENT));
+ boolean optional = jsonNode.get(ELEMENT_OPTIONAL).asBoolean();
+ return Types.ArrayType.get(elementId, optional, elementType);
+ } else if (MAP.equals(typeStr)) {
+ int keyId = jsonNode.get(KEY_ID).asInt();
+ Type keyType = parserTypeFromJson(jsonNode.get(KEY));
+ int valueId = jsonNode.get(VALUE_ID).asInt();
+ Type valueType = parserTypeFromJson(jsonNode.get(VALUE));
+ boolean optional = jsonNode.get(VALUE_OPTIONAL).asBoolean();
+ return Types.MapType.get(keyId, valueId, keyType, valueType, optional);
+ }
+ }
+ throw new IllegalArgumentException(String.format("cannot parse type from jsonNode: %s", jsonNode));
+ }
+
+ public static InternalSchema fromJson(JsonNode jsonNode) {
+ Integer maxColumnId = !jsonNode.has(MAX_COLUMN_ID) ? null : jsonNode.get(MAX_COLUMN_ID).asInt();
+ Long versionId = !jsonNode.has(VERSION_ID) ? null : jsonNode.get(VERSION_ID).asLong();
+ Types.RecordType type = (Types.RecordType)parserTypeFromJson(jsonNode);
+ if (versionId == null) {
+ return new InternalSchema(type.fields());
+ } else {
+ if (maxColumnId != null) {
+ return new InternalSchema(versionId, maxColumnId, type.fields());
+ } else {
+ return new InternalSchema(versionId, type.fields());
+ }
+ }
+ }
+
+ public static InternalSchema fromJson(String json) {
+ if (json == null || json.isEmpty()) {
+ return null;
+ }
+ try {
+ return fromJson((new ObjectMapper(new JsonFactory())).readValue(json, JsonNode.class));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static TreeMap<Long, InternalSchema> parserSchemas(String json) {
Review comment:
nit: Rename the method to `parseSchemas` (instead of parse**r**Schemas)
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
+ Integer nameId = querySchema.findIdByName(name);
+ if (nameId == null) {
+ throw new IllegalArgumentException(String.format("cannot found filter col name：%s from querySchema: %s", name, querySchema));
+ }
+ if (fileSchema.findField(nameId) == null) {
+ // added operation found
+ // the read file does not contain current col, so current colFilter is invalid
+ return "";
+ } else {
+ if (name.equals(fileSchema.findfullName(nameId))) {
Review comment:
This else block can be simplified to return `fileSchema.findfullName(nameId)` as we're doing the same thing in both the cases right?
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -87,4 +88,9 @@ trait SparkAdapter extends Serializable {
* Create Like expression.
*/
def createLike(left: Expression, right: Expression): Expression
+
+ /**
+ * Create parquet file read for hudi.
+ */
+ def createHoodieParquetFileFormat(): ParquetFileFormat
Review comment:
We need to call this for all table and query types not just MOR snapshot relation. So, basically at following places:
```
DefaultSource#getBaseFileOnlyView
IncrementalRelation
MergeOnReadIncrementalRelation
```
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/Types.java
##########
@@ -0,0 +1,716 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema;
+
+import org.apache.hudi.internal.schema.Type.PrimitiveType;
+import org.apache.hudi.internal.schema.Type.NestedType;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+public class Types {
+ private Types() {
+ }
+
+ public static class BooleanType extends PrimitiveType {
+ private static final BooleanType INSTANCE = new BooleanType();
+
+ public static BooleanType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return Type.TypeID.BOOLEAN;
+ }
+
+ @Override
+ public String toString() {
+ return "boolean";
+ }
+ }
+
+ public static class IntType extends PrimitiveType {
+ private static final IntType INSTANCE = new IntType();
+
+ public static IntType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.INT;
+ }
+
+ @Override
+ public String toString() {
+ return "int";
+ }
+ }
+
+ public static class LongType extends PrimitiveType {
+ private static final LongType INSTANCE = new LongType();
+
+ public static LongType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.LONG;
+ }
+
+ @Override
+ public String toString() {
+ return "long";
+ }
+ }
+
+ public static class FloatType extends PrimitiveType {
+ private static final FloatType INSTANCE = new FloatType();
+
+ public static FloatType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.FLOAT;
+ }
+
+ @Override
+ public String toString() {
+ return "float";
+ }
+ }
+
+ public static class DoubleType extends PrimitiveType {
+ private static final DoubleType INSTANCE = new DoubleType();
+
+ public static DoubleType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.DOUBLE;
+ }
+
+ @Override
+ public String toString() {
+ return "double";
+ }
+ }
+
+ public static class DateType extends PrimitiveType {
+ private static final DateType INSTANCE = new DateType();
+
+ public static DateType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.DATE;
+ }
+
+ @Override
+ public String toString() {
+ return "date";
+ }
+ }
+
+ public static class TimeType extends PrimitiveType {
+ private static final TimeType INSTANCE = new TimeType();
+
+ public static TimeType get() {
+ return INSTANCE;
+ }
+
+ private TimeType() {
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.TIME;
+ }
+
+ @Override
+ public String toString() {
+ return "time";
+ }
+ }
+
+ public static class TimestampType extends PrimitiveType {
+ private static final TimestampType INSTANCE = new TimestampType();
+
+ public static TimestampType get() {
+ return INSTANCE;
+ }
+
+ private TimestampType() {
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.TIMESTAMP;
+ }
+
+ @Override
+ public String toString() {
+ return "timestamp";
+ }
+ }
+
+ public static class StringType extends PrimitiveType {
+ private static final StringType INSTANCE = new StringType();
+
+ public static StringType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.STRING;
+ }
+
+ @Override
+ public String toString() {
+ return "string";
+ }
+ }
+
+ public static class BinaryType extends PrimitiveType {
+ private static final BinaryType INSTANCE = new BinaryType();
+
+ public static BinaryType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.BINARY;
+ }
+
+ @Override
+ public String toString() {
+ return "binary";
+ }
+ }
+
+ public static class FixedType extends PrimitiveType {
+ public static FixedType getFixed(int size) {
+ return new FixedType(size);
+ }
+
+ private final int size;
+
+ private FixedType(int length) {
+ this.size = length;
+ }
+
+ public int getFixedSize() {
+ return size;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.FIXED;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("fixed[%d]", size);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ } else if (!(o instanceof FixedType)) {
+ return false;
+ }
+
+ FixedType fixedType = (FixedType) o;
+ return size == fixedType.size;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(FixedType.class, size);
+ }
+ }
+
+ public static class DecimalType extends PrimitiveType {
+ public static DecimalType get(int precision, int scale) {
+ return new DecimalType(precision, scale);
+ }
+
+ private final int scale;
+ private final int precision;
+
+ private DecimalType(int precision, int scale) {
+ this.scale = scale;
+ this.precision = precision;
+ }
+
+ /**
+ * Returns whether this DecimalType is wider than `other`. If yes, it means `other`
+ * can be casted into `this` safely without losing any precision or range.
+ */
+ public boolean isWiderThan(PrimitiveType other) {
+ if (other instanceof DecimalType) {
+ DecimalType dt = (DecimalType) other;
+ return (precision - scale) >= (dt.precision - dt.scale) && scale > dt.scale;
+ }
+ if (other instanceof IntType) {
+ return isWiderThan(get(10, 0));
+ }
+ return false;
+ }
+
+ /**
+ * Returns whether this DecimalType is tighter than `other`. If yes, it means `this`
+ * can be casted into `other` safely without losing any precision or range.
+ */
+ public boolean isTighterThan(PrimitiveType other) {
+ if (other instanceof DecimalType) {
+ DecimalType dt = (DecimalType) other;
+ return (precision - scale) <= (dt.precision - dt.scale) && scale <= dt.scale;
+ }
+ if (other instanceof IntType) {
+ return isTighterThan(get(10, 0));
+ }
+ return false;
+ }
+
+ public int scale() {
+ return scale;
+ }
+
+ public int precision() {
+ return precision;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.DECIMAL;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("decimal(%d, %d)", precision, scale);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ } else if (!(o instanceof DecimalType)) {
+ return false;
+ }
+
+ DecimalType that = (DecimalType) o;
+ if (scale != that.scale) {
+ return false;
+ }
+ return precision == that.precision;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(DecimalType.class, scale, precision);
+ }
+ }
+
+ public static class UUIDType extends PrimitiveType {
+ private static final UUIDType INSTANCE = new UUIDType();
+
+ public static UUIDType get() {
+ return INSTANCE;
+ }
+
+ @Override
+ public TypeID typeId() {
+ return TypeID.UUID;
+ }
+
+ @Override
+ public String toString() {
+ return "uuid";
+ }
+ }
+
+ /** A field within a record. */
+ public static class Field implements Serializable {
+ // Experimental method to support defaultValue
+ public static Field get(int id, boolean isOptional, String name, Type type, String doc, Object defaultValue) {
+ return new Field(isOptional, id, name, type, doc, defaultValue);
+ }
+
+ public static Field get(int id, boolean isOptional, String name, Type type, String doc) {
+ return new Field(isOptional, id, name, type, doc, null);
+ }
+
+ public static Field get(int id, boolean isOptional, String name, Type type) {
+ return new Field(isOptional, id, name, type, null, null);
+ }
+
+ public static Field get(int id, String name, Type type) {
+ return new Field(true, id, name, type, null, null);
+ }
+
+ private final boolean isOptional;
+ private final int id;
+ private final String name;
+ private final Type type;
+ private final String doc;
+ // Experimental properties
+ private final Object defaultValue;
+
+ private Field(boolean isOptional, int id, String name, Type type, String doc, Object defaultValue) {
+ this.isOptional = isOptional;
+ this.id = id;
+ this.name = name;
+ this.type = type;
+ this.doc = doc;
+ this.defaultValue = defaultValue;
+ }
+
+ public Object getDefaultValue() {
+ return defaultValue;
+ }
+
+ public boolean isOptional() {
+ return isOptional;
+ }
+
+ public int fieldId() {
+ return id;
+ }
+
+ public String name() {
+ return name;
+ }
+
+ public Type type() {
+ return type;
+ }
+
+ public String doc() {
+ return doc;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("%d: %s: %s %s",
+ id, name, isOptional ? "optional" : "required", type) + (doc != null ? " (" + doc + ")" : "");
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ } else if (!(o instanceof Field)) {
+ return false;
+ }
+
+ Field that = (Field) o;
+ if (isOptional != that.isOptional) {
+ return false;
+ } else if (id != that.id) {
+ return false;
+ } else if (!name.equals(that.name)) {
+ return false;
+ } else if (!Objects.equals(doc, that.doc)) {
+ return false;
+ }
+ return type.equals(that.type);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(Field.class, id, isOptional, name, type);
+ }
+ }
+
+ public static class RecordType extends NestedType {
+
+ public static RecordType get(List<Field> fields) {
+ return new RecordType(fields);
+ }
+
+ public static RecordType get(Field... fields) {
+ return new RecordType(Arrays.asList(fields));
+ }
+
+ private final Field[] fields;
+
+ private transient Map<String, Field> nameToFields = null;
+ private transient Map<Integer, Field> idToFields = null;
+
+ private RecordType(List<Field> fields) {
+ this.fields = new Field[fields.size()];
+ for (int i = 0; i < this.fields.length; i += 1) {
+ this.fields[i] = fields.get(i);
+ }
+ }
+
+ @Override
+ public List<Field> fields() {
+ return Arrays.asList(fields);
+ }
+
+ public Field field(String name) {
+ if (nameToFields == null) {
+ nameToFields = new HashMap<>();
+ for (Field field : fields) {
+ nameToFields.put(field.name().toLowerCase(Locale.ROOT), field);
+ }
+ }
+ return nameToFields.get(name.toLowerCase(Locale.ROOT));
Review comment:
Let's do `getOrDefault` here and at other places to avoid potential NPE at caller site.
Optionally, we could add a Field.get() static method that returns empty field with no name and a negative id value that is never possible.
##########
File path: hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark2HoodieParquetFileFormat.scala
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.net.URI
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.mapreduce.lib.input.FileSplit
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.hadoop.mapreduce.{JobID, TaskAttemptID, TaskID, TaskType}
+import org.apache.hudi.client.utils.SparkSchemaUtils
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.util.TableInternalSchemaUtils
+import org.apache.hudi.internal.schema.InternalSchema
+import org.apache.hudi.internal.schema.utils.{InternalSchemaParser, InternalSchemaUtils}
+import org.apache.parquet.filter2.compat.FilterCompat
+import org.apache.parquet.filter2.predicate.FilterApi
+import org.apache.parquet.format.converter.ParquetMetadataConverter.SKIP_ROW_GROUPS
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetRecordReader}
+import org.apache.spark.TaskContext
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{JoinedRow, UnsafeRow}
+import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.execution.datasources.{PartitionedFile, RecordReaderIterator}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types.{AtomicType, StructType}
+import org.apache.spark.util.SerializableConfiguration
+
+class Spark2HoodieParquetFileFormat extends ParquetFileFormat {
+ // reference ParquetFileFormat from spark project
+ override def buildReaderWithPartitionValues(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ options: Map[String, String],
+ hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ if (hadoopConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA).isEmpty) {
+ // fallback to origin parquet File read
+ super.buildReaderWithPartitionValues(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf)
+ } else {
+ hadoopConf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ hadoopConf.set(
+ ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ ParquetWriteSupport.SPARK_ROW_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ SQLConf.SESSION_LOCAL_TIMEZONE.key,
+ sparkSession.sessionState.conf.sessionLocalTimeZone)
+ hadoopConf.setBoolean(
+ SQLConf.NESTED_SCHEMA_PRUNING_ENABLED.key,
+ sparkSession.sessionState.conf.nestedSchemaPruningEnabled)
+ hadoopConf.setBoolean(
+ SQLConf.CASE_SENSITIVE.key,
+ sparkSession.sessionState.conf.caseSensitiveAnalysis)
+
+ ParquetWriteSupport.setSchema(requiredSchema, hadoopConf)
+
+ // Sets flags for `ParquetToSparkSchemaConverter`
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_BINARY_AS_STRING.key,
+ sparkSession.sessionState.conf.isParquetBinaryAsString)
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP.key,
+ sparkSession.sessionState.conf.isParquetINT96AsTimestamp)
+
+ val broadcastedHadoopConf =
+ sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))
+
+ // TODO: if you move this into the closure it reverts to the default values.
+ // If true, enable using the custom RecordReader for parquet. This only works for
+ // a subset of the types (no complex types).
+ val resultSchema = StructType(partitionSchema.fields ++ requiredSchema.fields)
+ val sqlConf = sparkSession.sessionState.conf
+ val enableOffHeapColumnVector = sqlConf.offHeapColumnVectorEnabled
+ val enableVectorizedReader: Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ resultSchema.forall(_.dataType.isInstanceOf[AtomicType])
+ val enableRecordFilter: Boolean = sqlConf.parquetRecordFilterEnabled
+ val timestampConversion: Boolean = sqlConf.isParquetINT96TimestampConversion
+ val capacity = sqlConf.parquetVectorizedReaderBatchSize
+ val enableParquetFilterPushDown: Boolean = sqlConf.parquetFilterPushDown
+ // Whole stage codegen (PhysicalRDD) is able to deal with batches directly
+ val returningBatch = supportBatch(sparkSession, resultSchema)
+ val pushDownDate = sqlConf.parquetFilterPushDownDate
+ val pushDownTimestamp = sqlConf.parquetFilterPushDownTimestamp
+ val pushDownDecimal = sqlConf.parquetFilterPushDownDecimal
+ val pushDownStringStartWith = sqlConf.parquetFilterPushDownStringStartWith
+ val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
+ val isCaseSensitive = sqlConf.caseSensitiveAnalysis
+
+ (file: PartitionedFile) => {
+ assert(file.partitionValues.numFields == partitionSchema.size)
+
+ val fileSplit =
+ new FileSplit(new Path(new URI(file.filePath)), file.start, file.length, Array.empty)
+ val filePath = fileSplit.getPath
+
+ val split =
+ new org.apache.parquet.hadoop.ParquetInputSplit(
+ filePath,
+ fileSplit.getStart,
+ fileSplit.getStart + fileSplit.getLength,
+ fileSplit.getLength,
+ fileSplit.getLocations,
+ null)
+
+ val sharedConf = broadcastedHadoopConf.value.value
+
+ // do deal with internalSchema
+ val internalSchemaString = sharedConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA)
+ val internalSchemaChangeEnabled = if (internalSchemaString.isEmpty) false else true
+ // querySchema must be a pruned schema.
+ val querySchema = InternalSchemaParser.fromJson(internalSchemaString)
Review comment:
Corner case: If `internalSchemaString` is empty then `querySchema` could be null. Let's handle this.
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java
##########
@@ -0,0 +1,642 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import org.apache.avro.JsonProperties;
+import org.apache.avro.LogicalType;
+import org.apache.avro.LogicalTypes;
+import org.apache.avro.Schema;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+import org.apache.hudi.internal.schema.Types.Field;
+import org.apache.hudi.internal.schema.Types.RecordType;
+import org.apache.hudi.internal.schema.action.MergeSchemaAction;
+
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+public class InternalSchemaUtils {
+
+ private InternalSchemaUtils() {
+ }
+
+ public static Map<Integer, String> buildIdToName(Type type) {
+ Map<Integer, String> result = new HashMap<>();
+ buildNameToId(type).forEach((k, v) -> result.put(v, k));
+ return result;
+ }
+
+ public static Map<String, Integer> buildNameToId(Type type) {
+ Deque<String> fieldNames = new LinkedList<>();
+ Map<String, Integer> nameToId = new HashMap<>();
+ visitNameToId(type, fieldNames, nameToId);
+ return nameToId;
+ }
+
+ public static Map<Integer, Field> buildIdToField(Type type) {
+ Map<Integer, Field> idToField = new HashMap<>();
+ visitIdToField(type, idToField);
+ return idToField;
+ }
+
+ private static void visitIdToField(Type type, Map<Integer, Field> index) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ visitIdToField(field.type(), index);
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ visitIdToField(array.elementType(), index);
+ for (Field field : array.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ visitIdToField(map.keyType(), index);
+ visitIdToField(map.valueType(), index);
+ for (Field field : map.fields()) {
+ index.put(field.fieldId(), field);
+ }
+ return;
+ default:
+ return;
+ }
+ }
+
+ private static void visitNameToId(Type type, Deque<String> fieldNames, Map<String, Integer> nameToId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ for (Field field : record.fields()) {
+ fieldNames.push(field.name());
+ visitNameToId(field.type(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName(field.name(), fieldNames), field.fieldId());
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Field elementField = array.field(array.elementId());
+ fieldNames.push(elementField.name());
+ visitNameToId(array.elementType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("element", fieldNames), array.elementId());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Field keyField = map.field(map.keyId());
+ Field valueField = map.field(map.valueId());
+ fieldNames.push(keyField.name());
+ visitNameToId(map.keyType(), fieldNames, nameToId);
+ fieldNames.pop();
+ fieldNames.push(valueField.name());
+ visitNameToId(map.valueType(), fieldNames, nameToId);
+ fieldNames.pop();
+ nameToId.put(createFullName("key", fieldNames), map.keyId());
+ nameToId.put(createFullName("value", fieldNames), map.valueId());
+ return;
+ default:
+ }
+ }
+
+ public static String createFullName(String name, Deque<String> fieldNames) {
+ String result = name;
+ if (!fieldNames.isEmpty()) {
+ List<String> parentNames = new ArrayList<>();
+ fieldNames.descendingIterator().forEachRemaining(parentNames::add);
+ result = parentNames.stream().collect(Collectors.joining(".")) + "." + result;
+ }
+ return result;
+ }
+
+ public static Map<Integer, Integer> index2Parents(Types.RecordType record) {
+ Map<Integer, Integer> result = new HashMap<>();
+ Deque<Integer> parentIds = new LinkedList<>();
+ index2Parents(record, parentIds, result);
+ return result;
+ }
+
+ private static void index2Parents(Type type, Deque<Integer> pids, Map<Integer, Integer> id2p) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ for (Field f : record.fields()) {
+ pids.push(f.fieldId());
+ index2Parents(f.type(), pids, id2p);
+ pids.pop();
+ }
+
+ for (Field f : record.fields()) {
+ // root record has no parent id.
+ if (!pids.isEmpty()) {
+ Integer pid = pids.peek();
+ id2p.put(f.fieldId(), pid);
+ }
+ }
+ return;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Types.Field elementField = array.field(array.elementId());
+ pids.push(elementField.fieldId());
+ index2Parents(elementField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(array.elementId(), pids.peek());
+ return;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Types.Field keyField = map.field(map.keyId());
+ Types.Field valueField = map.field(map.valueId());
+ // visit key
+ pids.push(map.keyId());
+ index2Parents(keyField.type(), pids, id2p);
+ pids.pop();
+ // visit value
+ pids.push(map.valueId());
+ index2Parents(valueField.type(), pids, id2p);
+ pids.pop();
+ id2p.put(map.keyId(), pids.peek());
+ id2p.put(map.valueId(), pids.peek());
+ return;
+ default:
+ }
+ }
+
+ public static Type refreshNewId(Type type, AtomicInteger nextId) {
+ switch (type.typeId()) {
+ case RECORD:
+ RecordType record = (RecordType) type;
+ List<Field> oldFields = record.fields();
+ List<Type> fieldTypes = new ArrayList<>();
+ int currentId = nextId.get();
+ nextId.set(currentId + record.fields().size());
+ for (Types.Field f : oldFields) {
+ fieldTypes.add(refreshNewId(f.type(), nextId));
+ }
+ List<Types.Field> internalFields = new ArrayList<>();
+ for (int i = 0; i < oldFields.size(); i++) {
+ Field oldField = oldFields.get(i);
+ internalFields.add(Types.Field.get(currentId++, oldField.isOptional(), oldField.name(), fieldTypes.get(i), oldField.doc()));
+ }
+ return Types.RecordType.get(internalFields);
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = refreshNewId(array.elementType(), nextId);
+ return Types.ArrayType.get(elementId, array.isElementOptional(), elementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(keyId + 2);
+ Type keyType = refreshNewId(map.keyType(), nextId);
+ Type valueType = refreshNewId(map.valueType(), nextId);
+ return Types.MapType.get(keyId, valueId, keyType, valueType, map.isValueOptional());
+ default:
+ return type;
+ }
+ }
+
+ public static Type buildTypeFromAvroSchema(Schema schema) {
+ // set flag to check this has not been visited.
+ Deque<String> visited = new LinkedList();
+ Deque<String> fieldNames = new LinkedList();
+ AtomicInteger nextId = new AtomicInteger(1);
+ return visitAvroSchemaToBuildType(schema, visited, fieldNames, true, nextId);
+ }
+
+ private static Type visitAvroSchemaToBuildType(Schema schema, Deque<String> visited, Deque<String> fieldNames, Boolean firstVisitRoot, AtomicInteger nextId) {
+ switch (schema.getType()) {
+ case RECORD:
+ String name = schema.getFullName();
+ if (visited.contains(name)) {
+ throw new HoodieSchemaException(String.format("cannot convert recursive avro record %s", name));
+ }
+ visited.push(name);
+ List<Schema.Field> fields = schema.getFields();
+ List<Type> fieldTypes = new ArrayList<>(fields.size());
+ int nextAssignId = nextId.get();
+ // when first visit root record, set nextAssignId = 0;
+ if (firstVisitRoot) {
+ nextAssignId = 0;
+ }
+ nextId.set(nextAssignId + fields.size());
+ fields.stream().forEach(field -> {
+ fieldNames.addLast(name);
+ fieldTypes.add(visitAvroSchemaToBuildType(field.schema(), visited, fieldNames, false, nextId));
+ });
+ visited.pop();
+ List<Types.Field> internalFields = new ArrayList<>(fields.size());
+
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field field = fields.get(i);
+ Type fieldType = fieldTypes.get(i);
+ internalFields.add(Types.Field.get(nextAssignId, AvroSchemaUtil.isOptional(field.schema()), field.name(), fieldType, field.doc()));
+ nextAssignId += 1;
+ }
+ return Types.RecordType.get(internalFields);
+ case UNION:
+ List<Type> fTypes = new ArrayList<>();
+ schema.getTypes().stream().forEach(t -> {
+ fTypes.add(visitAvroSchemaToBuildType(t, visited, fieldNames, false, nextId));
+ });
+ return fTypes.get(0) == null ? fTypes.get(1) : fTypes.get(0);
+ case ARRAY:
+ Schema elementSchema = schema.getElementType();
+ fieldNames.addLast("element");
+ int elementId = nextId.get();
+ nextId.set(elementId + 1);
+ Type elementType = visitAvroSchemaToBuildType(elementSchema, visited, fieldNames, false, nextId);
+ fieldNames.pop();
+ return Types.ArrayType.get(elementId, AvroSchemaUtil.isOptional(schema.getElementType()), elementType);
+ case MAP:
+ fieldNames.addLast("value");
+ int keyId = nextId.get();
+ int valueId = keyId + 1;
+ nextId.set(valueId + 1);
+ Type valueType = visitAvroSchemaToBuildType(schema.getValueType(), visited, fieldNames, false, nextId);
+ return Types.MapType.get(keyId, valueId, Types.StringType.get(), valueType, AvroSchemaUtil.isOptional(schema.getValueType()));
+ default:
+ return visitAvroPrimitiveToBuildInternalType(schema);
+ }
+ }
+
+ private static Type visitAvroPrimitiveToBuildInternalType(Schema primitive) {
+ LogicalType logical = primitive.getLogicalType();
+ if (logical != null) {
+ String name = logical.getName();
+ if (logical instanceof LogicalTypes.Decimal) {
+ return Types.DecimalType.get(
+ ((LogicalTypes.Decimal) logical).getPrecision(),
+ ((LogicalTypes.Decimal) logical).getScale());
+
+ } else if (logical instanceof LogicalTypes.Date) {
+ return Types.DateType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimeMillis
+ || logical instanceof LogicalTypes.TimeMicros) {
+ return Types.TimeType.get();
+
+ } else if (
+ logical instanceof LogicalTypes.TimestampMillis
+ || logical instanceof LogicalTypes.TimestampMicros) {
+ return Types.TimestampType.get();
+ } else if (LogicalTypes.uuid().getName().equals(name)) {
+ return Types.UUIDType.get();
+ }
+ }
+
+ switch (primitive.getType()) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case STRING:
+ case ENUM:
+ return Types.StringType.get();
+ case FIXED:
+ return Types.FixedType.getFixed(primitive.getFixedSize());
+ case BYTES:
+ return Types.BinaryType.get();
+ case NULL:
+ return null;
+ default:
+ throw new UnsupportedOperationException("Unsupported primitive type: " + primitive);
+ }
+ }
+
+ public static Schema buildAvroSchemaFromType(Type type, String name) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(type, cache, name);
+ }
+
+ public static Schema buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) {
+ Map<Type, Schema> cache = new HashMap<>();
+ return visitInternalSchemaToBuildAvroSchema(schema.getRecord(), cache, recordName);
+ }
+
+ private static Schema visitInternalSchemaToBuildAvroSchema(Type type, Map<Type, Schema> cache, String recordName) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Schema> schemas = new ArrayList<>();
+ record.fields().forEach(f -> {
+ Schema tempSchema = visitInternalSchemaToBuildAvroSchema(f.type(), cache, recordName + "_" + f.name());
+ // convert tempSchema
+ Schema result = f.isOptional() ? AvroSchemaUtil.nullableSchema(tempSchema) : tempSchema;
+ schemas.add(result);
+ });
+ // check visited
+ Schema recordSchema;
+ recordSchema = cache.get(record);
+ if (recordSchema != null) {
+ return recordSchema;
+ }
+ recordSchema = visitInternalRecordToBuildAvroRecord(record, schemas, recordName);
+ cache.put(record, recordSchema);
+ return recordSchema;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Schema elementSchema;
+ elementSchema = visitInternalSchemaToBuildAvroSchema(array.elementType(), cache, recordName);
+ Schema arraySchema;
+ arraySchema = cache.get(array);
+ if (arraySchema != null) {
+ return arraySchema;
+ }
+ arraySchema = visitInternalArrayToBuildAvroArray(array, elementSchema);
+ cache.put(array, arraySchema);
+ return arraySchema;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Schema keySchema;
+ Schema valueSchema;
+ keySchema = visitInternalSchemaToBuildAvroSchema(map.keyType(), cache, recordName);
+ valueSchema = visitInternalSchemaToBuildAvroSchema(map.valueType(), cache, recordName);
+ Schema mapSchema;
+ mapSchema = cache.get(map);
+ if (mapSchema != null) {
+ return mapSchema;
+ }
+ mapSchema = visitInternalMapToBuildAvroMap(map, keySchema, valueSchema);
+ cache.put(map, mapSchema);
+ return mapSchema;
+ default:
+ Schema primitiveSchema = visitInternalPrimitiveToBuildAvroMap((Type.PrimitiveType) type);
+ cache.put(type, primitiveSchema);
+ return primitiveSchema;
+ }
+ }
+
+ private static Schema visitInternalRecordToBuildAvroRecord(Types.RecordType record, List<Schema> fieldSchemas, String recordName) {
+ List<Field> fields = record.fields();
+ List<Schema.Field> avroFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Field f = fields.get(i);
+ Schema.Field field = new Schema.Field(f.name(), fieldSchemas.get(i), f.doc(), f.isOptional() ? JsonProperties.NULL_VALUE : null);
+ avroFields.add(field);
+ }
+ return Schema.createRecord(recordName, null, null, false, avroFields);
+ }
+
+ private static Schema visitInternalArrayToBuildAvroArray(Types.ArrayType array, Schema elementSchema) {
+ Schema result;
+ if (array.isElementOptional()) {
+ result = Schema.createArray(AvroSchemaUtil.nullableSchema(elementSchema));
+ } else {
+ result = Schema.createArray(elementSchema);
+ }
+ return result;
+ }
+
+ private static Schema visitInternalMapToBuildAvroMap(Types.MapType map, Schema keySchema, Schema valueSchema) {
+ Schema mapSchema;
+ if (keySchema.getType() == Schema.Type.STRING) {
+ mapSchema = Schema.createMap(map.isValueOptional() ? AvroSchemaUtil.nullableSchema(valueSchema) : valueSchema);
+ } else {
+ throw new HoodieSchemaException("only support StringType key for avro MapType");
+ }
+ return mapSchema;
+ }
+
+ private static Schema visitInternalPrimitiveToBuildAvroMap(Type.PrimitiveType primitive) {
+ Schema primitiveSchema;
+ switch (primitive.typeId()) {
+ case BOOLEAN:
+ primitiveSchema = Schema.create(Schema.Type.BOOLEAN);
+ break;
+ case INT:
+ primitiveSchema = Schema.create(Schema.Type.INT);
+ break;
+ case LONG:
+ primitiveSchema = Schema.create(Schema.Type.LONG);
+ break;
+ case FLOAT:
+ primitiveSchema = Schema.create(Schema.Type.FLOAT);
+ break;
+ case DOUBLE:
+ primitiveSchema = Schema.create(Schema.Type.DOUBLE);
+ break;
+ case DATE:
+ primitiveSchema = LogicalTypes.date()
+ .addToSchema(Schema.create(Schema.Type.INT));
+ break;
+ case TIME:
+ primitiveSchema = LogicalTypes.timeMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case TIMESTAMP:
+ primitiveSchema = LogicalTypes.timestampMicros()
+ .addToSchema(Schema.create(Schema.Type.LONG));
+ break;
+ case STRING:
+ primitiveSchema = Schema.create(Schema.Type.STRING);
+ break;
+ case UUID:
+ primitiveSchema = LogicalTypes.uuid()
+ .addToSchema(Schema.createFixed("uuid_fixed", null, null, 16));
+ break;
+ case FIXED:
+ Types.FixedType fixed = (Types.FixedType) primitive;
+ primitiveSchema = Schema.createFixed("fixed_" + fixed.getFixedSize(), null, null, fixed.getFixedSize());
+ break;
+ case BINARY:
+ primitiveSchema = Schema.create(Schema.Type.BYTES);
+ break;
+ case DECIMAL:
+ Types.DecimalType decimal = (Types.DecimalType) primitive;
+ primitiveSchema = LogicalTypes.decimal(decimal.precision(), decimal.scale())
+ .addToSchema(Schema.createFixed(
+ "decimal_" + decimal.precision() + "_" + decimal.scale(),
+ null, null, computeMinBytesForPrecision(decimal.precision())));
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported type ID: " + primitive.typeId());
+ }
+ return primitiveSchema;
+ }
+
+ private static int computeMinBytesForPrecision(int precision) {
+ int numBytes = 1;
+ while (Math.pow(2.0, 8 * numBytes - 1) < Math.pow(10.0, precision)) {
+ numBytes += 1;
+ }
+ return numBytes;
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema) {
+ return mergeSchema(fileSchema, querySchema, true);
+ }
+
+ public static InternalSchema mergeSchema(InternalSchema fileSchema, InternalSchema querySchema, Boolean mergeRequiredFiledForce) {
+ MergeSchemaAction mergeSchemaAction = new MergeSchemaAction(fileSchema, querySchema, mergeRequiredFiledForce);
+ Types.RecordType record = (Types.RecordType) mergeType(mergeSchemaAction, querySchema.getRecord());
+ return new InternalSchema(record.fields());
+ }
+
+ private static Type mergeType(MergeSchemaAction mergeSchemaAction, Type type) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : record.fields()) {
+ Type newType = mergeType(mergeSchemaAction, f.type());
+ newTypes.add(newType);
+ }
+ return Types.RecordType.get(mergeSchemaAction.buildNewFields(record.fields(), newTypes));
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType;
+ Types.Field elementField = array.fields().get(0);
+ newElementType = mergeType(mergeSchemaAction, elementField.type());
+ return mergeSchemaAction.buildArrayType(array, newElementType);
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = mergeType(mergeSchemaAction, map.valueType());
+ return mergeSchemaAction.buildMapType(map, newValueType);
+ default:
+ return type;
+ }
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<String> names) {
+ // do check
+ HashSet prunedIds = names.stream().map(name -> schema.findIdByName(name.toLowerCase(Locale.ROOT)))
+ .collect(Collectors.toCollection(HashSet::new));
+ if (prunedIds.contains(-1)) {
+ throw new IllegalArgumentException("cannot prune col which not exisit in hudi table");
+ }
+ return pruneInternalSchema(schema, prunedIds);
+ }
+
+ public static InternalSchema pruneInternalSchema(InternalSchema schema, Set fieldIds) {
+ Types.RecordType recordType = (Types.RecordType)pruneType(schema.getRecord(), fieldIds);
+ return new InternalSchema(recordType.fields());
+ }
+
+ private static Type pruneType(Type type, Set<Integer> fieldIds) {
+ switch (type.typeId()) {
+ case RECORD:
+ Types.RecordType record = (Types.RecordType) type;
+ List<Types.Field> fields = record.fields();
+ List<Type> newTypes = new ArrayList<>();
+ for (Types.Field f : fields) {
+ Type newType = pruneType(f.type(), fieldIds);
+ if (fieldIds.contains(f.fieldId())) {
+ newTypes.add(f.type());
+ } else if (newType != null) {
+ newTypes.add(newType);
+ } else {
+ newTypes.add(null);
+ }
+ }
+ boolean changed = false;
+ List<Field> newFields = new ArrayList<>();
+ for (int i = 0; i < fields.size(); i++) {
+ Types.Field oldField = fields.get(i);
+ Type newType = newTypes.get(i);
+ if (oldField.type() == newType) {
+ newFields.add(oldField);
+ } else if (newType != null) {
+ changed = true;
+ newFields.add(Types.Field.get(oldField.fieldId(), oldField.isOptional(), oldField.name(), newType, oldField.doc()));
+ }
+ }
+ if (newFields.isEmpty()) {
+ return null;
+ }
+ if (newFields.size() == fields.size() && !changed) {
+ return record;
+ } else {
+ return Types.RecordType.get(newFields);
+ }
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ Type newElementType = pruneType(array.elementType(), fieldIds);
+ if (fieldIds.contains(array.elementId())) {
+ return array;
+ } else if (newElementType != null) {
+ if (array.elementType() == newElementType) {
+ return array;
+ }
+ return Types.ArrayType.get(array.elementId(), array.isElementOptional(), newElementType);
+ }
+ return null;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ Type newValueType = pruneType(map.valueType(), fieldIds);
+ if (fieldIds.contains(map.valueId())) {
+ return map;
+ } else if (newValueType != null) {
+ if (map.valueType() == newValueType) {
+ return map;
+ }
+ return Types.MapType.get(map.keyId(), map.valueId(), map.keyType(), newValueType, map.isValueOptional());
+ }
+ return null;
+ default:
+ return null;
+ }
+ }
+
+ public static String reBuildFilterName(String name, InternalSchema fileSchema, InternalSchema querySchema) {
+ Integer nameId = querySchema.findIdByName(name);
Review comment:
Let's keep the primitive type to avoid unnecessary autoboxing/unboxing.
##########
File path: hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark2HoodieParquetFileFormat.scala
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.net.URI
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.mapreduce.lib.input.FileSplit
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.apache.hadoop.mapreduce.{JobID, TaskAttemptID, TaskID, TaskType}
+import org.apache.hudi.client.utils.SparkSchemaUtils
+import org.apache.hudi.common.fs.FSUtils
+import org.apache.hudi.common.util.TableInternalSchemaUtils
+import org.apache.hudi.internal.schema.InternalSchema
+import org.apache.hudi.internal.schema.utils.{InternalSchemaParser, InternalSchemaUtils}
+import org.apache.parquet.filter2.compat.FilterCompat
+import org.apache.parquet.filter2.predicate.FilterApi
+import org.apache.parquet.format.converter.ParquetMetadataConverter.SKIP_ROW_GROUPS
+import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetRecordReader}
+import org.apache.spark.TaskContext
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{JoinedRow, UnsafeRow}
+import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
+import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.execution.datasources.{PartitionedFile, RecordReaderIterator}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources._
+import org.apache.spark.sql.types.{AtomicType, StructType}
+import org.apache.spark.util.SerializableConfiguration
+
+class Spark2HoodieParquetFileFormat extends ParquetFileFormat {
+ // reference ParquetFileFormat from spark project
+ override def buildReaderWithPartitionValues(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ options: Map[String, String],
+ hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ if (hadoopConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA).isEmpty) {
+ // fallback to origin parquet File read
+ super.buildReaderWithPartitionValues(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf)
+ } else {
+ hadoopConf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
+ hadoopConf.set(
+ ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ ParquetWriteSupport.SPARK_ROW_SCHEMA,
+ requiredSchema.json)
+ hadoopConf.set(
+ SQLConf.SESSION_LOCAL_TIMEZONE.key,
+ sparkSession.sessionState.conf.sessionLocalTimeZone)
+ hadoopConf.setBoolean(
+ SQLConf.NESTED_SCHEMA_PRUNING_ENABLED.key,
+ sparkSession.sessionState.conf.nestedSchemaPruningEnabled)
+ hadoopConf.setBoolean(
+ SQLConf.CASE_SENSITIVE.key,
+ sparkSession.sessionState.conf.caseSensitiveAnalysis)
+
+ ParquetWriteSupport.setSchema(requiredSchema, hadoopConf)
+
+ // Sets flags for `ParquetToSparkSchemaConverter`
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_BINARY_AS_STRING.key,
+ sparkSession.sessionState.conf.isParquetBinaryAsString)
+ hadoopConf.setBoolean(
+ SQLConf.PARQUET_INT96_AS_TIMESTAMP.key,
+ sparkSession.sessionState.conf.isParquetINT96AsTimestamp)
+
+ val broadcastedHadoopConf =
+ sparkSession.sparkContext.broadcast(new SerializableConfiguration(hadoopConf))
+
+ // TODO: if you move this into the closure it reverts to the default values.
+ // If true, enable using the custom RecordReader for parquet. This only works for
+ // a subset of the types (no complex types).
+ val resultSchema = StructType(partitionSchema.fields ++ requiredSchema.fields)
+ val sqlConf = sparkSession.sessionState.conf
+ val enableOffHeapColumnVector = sqlConf.offHeapColumnVectorEnabled
+ val enableVectorizedReader: Boolean =
+ sqlConf.parquetVectorizedReaderEnabled &&
+ resultSchema.forall(_.dataType.isInstanceOf[AtomicType])
+ val enableRecordFilter: Boolean = sqlConf.parquetRecordFilterEnabled
+ val timestampConversion: Boolean = sqlConf.isParquetINT96TimestampConversion
+ val capacity = sqlConf.parquetVectorizedReaderBatchSize
+ val enableParquetFilterPushDown: Boolean = sqlConf.parquetFilterPushDown
+ // Whole stage codegen (PhysicalRDD) is able to deal with batches directly
+ val returningBatch = supportBatch(sparkSession, resultSchema)
+ val pushDownDate = sqlConf.parquetFilterPushDownDate
+ val pushDownTimestamp = sqlConf.parquetFilterPushDownTimestamp
+ val pushDownDecimal = sqlConf.parquetFilterPushDownDecimal
+ val pushDownStringStartWith = sqlConf.parquetFilterPushDownStringStartWith
+ val pushDownInFilterThreshold = sqlConf.parquetFilterPushDownInFilterThreshold
+ val isCaseSensitive = sqlConf.caseSensitiveAnalysis
+
+ (file: PartitionedFile) => {
+ assert(file.partitionValues.numFields == partitionSchema.size)
+
+ val fileSplit =
+ new FileSplit(new Path(new URI(file.filePath)), file.start, file.length, Array.empty)
+ val filePath = fileSplit.getPath
+
+ val split =
+ new org.apache.parquet.hadoop.ParquetInputSplit(
+ filePath,
+ fileSplit.getStart,
+ fileSplit.getStart + fileSplit.getLength,
+ fileSplit.getLength,
+ fileSplit.getLocations,
+ null)
+
+ val sharedConf = broadcastedHadoopConf.value.value
+
+ // do deal with internalSchema
+ val internalSchemaString = sharedConf.get(SparkSchemaUtils.HOODIE_QUERY_SCHEMA)
+ val internalSchemaChangeEnabled = if (internalSchemaString.isEmpty) false else true
+ // querySchema must be a pruned schema.
+ val querySchema = InternalSchemaParser.fromJson(internalSchemaString)
+ val tablePath = sharedConf.get(SparkSchemaUtils.HOODIE_TABLE_PATH)
+ val commitTime = FSUtils.getCommitTime(filePath.getName).toLong;
+ val fileSchema = if (internalSchemaChangeEnabled) {
+ TableInternalSchemaUtils.searchSchemaAndCache(commitTime, tablePath, sharedConf)
+ } else {
+ null
+ }
+
+ lazy val footerFileMetaData =
+ ParquetFileReader.readFooter(sharedConf, filePath, SKIP_ROW_GROUPS).getFileMetaData
+ // Try to push down filters when filter push-down is enabled.
+ val pushed = if (enableParquetFilterPushDown) {
+ val parquetSchema = footerFileMetaData.getSchema
+ val parquetFilters = new ParquetFilters(pushDownDate, pushDownTimestamp, pushDownDecimal,
+ pushDownStringStartWith, pushDownInFilterThreshold, isCaseSensitive)
+ filters.map(Spark2HoodieParquetFileFormat.rebuildFilterFromParquet(_, fileSchema, querySchema))
+ // Collects all converted Parquet filter predicates. Notice that not all predicates can be
+ // converted (`ParquetFilters.createFilter` returns an `Option`). That's why a `flatMap`
+ // is used here.
+ .flatMap(parquetFilters.createFilter(parquetSchema, _))
+ .reduceOption(FilterApi.and)
+ } else {
+ None
+ }
+
+ // PARQUET_INT96_TIMESTAMP_CONVERSION says to apply timezone conversions to int96 timestamps'
+ // *only* if the file was created by something other than "parquet-mr", so check the actual
+ // writer here for this file. We have to do this per-file, as each file in the table may
+ // have different writers.
+ // Define isCreatedByParquetMr as function to avoid unnecessary parquet footer reads.
+ def isCreatedByParquetMr: Boolean =
+ footerFileMetaData.getCreatedBy().startsWith("parquet-mr")
+
+ val convertTz =
+ if (timestampConversion && !isCreatedByParquetMr) {
+ Some(DateTimeUtils.getTimeZone(sharedConf.get(SQLConf.SESSION_LOCAL_TIMEZONE.key)))
+ } else {
+ None
+ }
+
+ val attemptId = new TaskAttemptID(new TaskID(new JobID(), TaskType.MAP, 0), 0)
+ val hadoopAttempConf = broadcastedHadoopConf.value.value
+ // reset request schema
+ if (internalSchemaChangeEnabled) {
+ val mergedSchema = SparkSchemaUtils.mergeSchema(fileSchema, querySchema)
+ hadoopAttempConf.set(ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA, mergedSchema.json)
+ }
+ val hadoopAttemptContext =
+ new TaskAttemptContextImpl(hadoopAttempConf, attemptId)
+
+ // Try to push down filters when filter push-down is enabled.
+ // Notice: This push-down is RowGroups level, not individual records.
+ if (pushed.isDefined) {
+ ParquetInputFormat.setFilterPredicate(hadoopAttemptContext.getConfiguration, pushed.get)
+ }
+ val taskContext = Option(TaskContext.get())
+ if (enableVectorizedReader) {
+ val vectorizedReader = new VectorizedParquetRecordReader(
+ convertTz.orNull, enableOffHeapColumnVector && taskContext.isDefined, capacity)
+ val iter = new RecordReaderIterator(vectorizedReader)
+ // SPARK-23457 Register a task completion lister before `initialization`.
+ taskContext.foreach(_.addTaskCompletionListener[Unit](_ => iter.close()))
+ vectorizedReader.initialize(split, hadoopAttemptContext)
+ logDebug(s"Appending $partitionSchema ${file.partitionValues}")
+ vectorizedReader.initBatch(partitionSchema, file.partitionValues)
+ if (returningBatch) {
+ vectorizedReader.enableReturningBatches()
+ }
+
+ // UnsafeRowParquetRecordReader appends the columns internally to avoid another copy.
+ iter.asInstanceOf[Iterator[InternalRow]]
+ } else {
+ logDebug(s"Falling back to parquet-mr")
+ // ParquetRecordReader returns UnsafeRow
+ val reader = if (pushed.isDefined && enableRecordFilter) {
+ val parquetFilter = FilterCompat.get(pushed.get, null)
+ new ParquetRecordReader[UnsafeRow](new ParquetReadSupport(convertTz), parquetFilter)
+ } else {
+ new ParquetRecordReader[UnsafeRow](new ParquetReadSupport(convertTz))
+ }
+ val iter = new RecordReaderIterator(reader)
+ // SPARK-23457 Register a task completion lister before `initialization`.
+ taskContext.foreach(_.addTaskCompletionListener[Unit](_ => iter.close()))
+ reader.initialize(split, hadoopAttemptContext)
+
+ val fullSchema = requiredSchema.toAttributes ++ partitionSchema.toAttributes
+ val joinedRow = new JoinedRow()
+ val appendPartitionColumns = GenerateUnsafeProjection.generate(fullSchema, fullSchema)
+
+ // This is a horrible erasure hack... if we type the iterator above, then it actually check
+ // the type in next() and we get a class cast exception. If we make that function return
+ // Object, then we can defer the cast until later!
+ if (partitionSchema.length == 0) {
+ // There is no partition columns
+ iter.asInstanceOf[Iterator[InternalRow]]
+ } else {
+ iter.asInstanceOf[Iterator[InternalRow]]
+ .map(d => appendPartitionColumns(joinedRow(d, file.partitionValues)))
+ }
+ }
+ }
+ }
+ }
+}
+
+object Spark2HoodieParquetFileFormat {
+
+ def rebuildFilterFromParquet(oldFilter: Filter, fileSchema: InternalSchema, querySchema: InternalSchema): Filter = {
Review comment:
We can keep this method private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r710898242
##########
File path: hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaParser.java
##########
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.internal.schema.utils;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.util.TokenBuffer;
+
+import org.apache.avro.JsonProperties;
+import org.apache.hadoop.hbase.exceptions.IllegalArgumentIOException;
+import org.apache.hudi.internal.schema.HoodieSchemaException;
+import org.apache.hudi.internal.schema.InternalSchema;
+import org.apache.hudi.internal.schema.Type;
+import org.apache.hudi.internal.schema.Types;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class InternalSchemaParser {
+ private InternalSchemaParser() {
+
+ }
+
+ public static final String LATESTSCHEMA = "latestSchema";
+ public static final String SCHEMAS = "schemas";
+ private static final String MAX_COLUMN_ID = "max_column_id";
+ private static final String VERSION_ID = "version-id";
+ private static final String TYPE = "type";
+ private static final String RECORD = "record";
+ private static final String ARRAY = "array";
+ private static final String MAP = "map";
+ private static final String FIELDS = "fields";
+ private static final String ELEMENT = "element";
+ private static final String KEY = "key";
+ private static final String VALUE = "value";
+ private static final String DOC = "doc";
+ private static final String NAME = "name";
+ private static final String ID = "id";
+ private static final String ELEMENT_ID = "element-id";
+ private static final String KEY_ID = "key-id";
+ private static final String VALUE_ID = "value-id";
+ private static final String OPTIONAL = "optional";
+ private static final String ELEMENT_OPTIONAL = "element_optional";
+ private static final String VALUE_OPTIONAL = "value_optional";
+
+ private static final Pattern FIXED = Pattern.compile("fixed\\[(\\d+)\\]");
+ private static final Pattern DECIMAL = Pattern.compile("decimal\\((\\d+),\\s+(\\d+)\\)");
+
+ public static JsonNode toJsonNode(Object datum) {
+ if (datum == null) {
+ return null;
+ }
+ try {
+ TokenBuffer generator = new TokenBuffer(new ObjectMapper(), false);
+ toJson(datum, generator);
+ return new ObjectMapper().readTree(generator.asParser());
+ } catch (IOException e) {
+ throw new HoodieSchemaException(e);
+ }
+ }
+
+ private static void toJson(Object datum, JsonGenerator generator) throws IOException {
+ if (datum == JsonProperties.NULL_VALUE) { // null
+ generator.writeNull();
+ } else if (datum instanceof Map) { // record, map
+ generator.writeStartObject();
+ for (Map.Entry<Object, Object> entry : ((Map<Object, Object>) datum).entrySet()) {
+ generator.writeFieldName(entry.getKey().toString());
+ toJson(entry.getValue(), generator);
+ }
+ generator.writeEndObject();
+ } else if (datum instanceof Collection) { // array
+ generator.writeStartArray();
+ for (Object element : (Collection) datum) {
+ toJson(element, generator);
+ }
+ generator.writeEndArray();
+ } else if (datum instanceof byte[]) { // bytes, fixed
+ generator.writeString(new String((byte[]) datum, StandardCharsets.ISO_8859_1));
+ } else if (datum instanceof CharSequence || datum instanceof Enum) { // string, enum
+ generator.writeString(datum.toString());
+ } else if (datum instanceof Double) { // double
+ generator.writeNumber((Double) datum);
+ } else if (datum instanceof Float) { // float
+ generator.writeNumber((Float) datum);
+ } else if (datum instanceof Long) { // long
+ generator.writeNumber((Long) datum);
+ } else if (datum instanceof Integer) { // int
+ generator.writeNumber((Integer) datum);
+ } else if (datum instanceof Boolean) { // boolean
+ generator.writeBoolean((Boolean) datum);
+ } else {
+ throw new HoodieSchemaException("Unknown datum class: " + datum.getClass());
+ }
+ }
+
+ public boolean isValidDefaultValue(Type type, JsonNode defaultValue) {
+ if (defaultValue == null) {
+ return true;
+ }
+ switch (type.typeId()) {
+ case STRING:
+ case BINARY:
+ case FIXED:
+ return defaultValue.isTextual();
+ case INT:
+ case LONG:
+ case FLOAT:
+ case DOUBLE:
+ return defaultValue.isNumber();
+ case BOOLEAN:
+ return defaultValue.isBoolean();
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType)type;
+ if (!defaultValue.isArray()) {
+ return false;
+ }
+ for (JsonNode element : defaultValue) {
+ if (!isValidDefaultValue(array.elementType(), element)) {
+ return false;
+ }
+ }
+ return true;
+ case MAP:
+ Types.MapType map = (Types.MapType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (JsonNode value : defaultValue) {
+ if (!isValidDefaultValue(map.valueType(), value)) {
+ return false;
+ }
+ }
+ return true;
+ case RECORD:
+ Types.RecordType record = (Types.RecordType)type;
+ if (!defaultValue.isObject()) {
+ return false;
+ }
+ for (Types.Field f : record.fields()) {
+ if (!isValidDefaultValue(f.type(), defaultValue.has(f.name()) ? defaultValue.get(f.name()) : toJsonNode(f.getDefaultValue()))) {
+ return false;
+ }
+ }
+ return true;
+ default:
+ return false;
+ }
+ }
+
+ public static Object toObject(JsonNode jsonNode, Type type) {
+ if (jsonNode == null) {
+ return null;
+ } else if (jsonNode.isNull()) {
+ return null;
+ } else if (jsonNode.isBoolean()) {
+ return jsonNode.asBoolean();
+ } else if (jsonNode.isInt()) {
+ if (type == null || type.typeId().equals(Type.TypeID.INT)) {
+ return jsonNode.asInt();
+ } else if (type.typeId().equals(Type.TypeID.LONG)) {
+ return jsonNode.asLong();
+ }
+ } else if (jsonNode.isLong()) {
+ return jsonNode.asLong();
+ } else if (jsonNode.isDouble() || jsonNode.isFloat()) {
+ if (type == null || type.typeId().equals(Type.TypeID.DOUBLE)) {
+ return jsonNode.asDouble();
+ } else if (type.typeId().equals(Type.TypeID.FLOAT)) {
+ return (float) jsonNode.asDouble();
+ }
+ } else if (jsonNode.isTextual()) {
+ if (type == null || type.typeId().equals(Type.TypeID.STRING)) {
+ return jsonNode.asText();
+ } else if (type.typeId().equals(Type.TypeID.BINARY) || type.typeId().equals(Type.TypeID.FIXED)) {
+ return jsonNode.textValue().getBytes(StandardCharsets.ISO_8859_1);
+ }
+ } else if (jsonNode.isArray()) {
+ List<Object> l = new ArrayList<>();
+ for (JsonNode node : jsonNode) {
+ l.add(toObject(node, type == null ? null : ((Types.ArrayType)type).elementType()));
+ }
+ return l;
+ } else if (jsonNode.isObject()) {
+ Map<Object, Object> m = new LinkedHashMap<>();
+ for (Iterator<String> it = jsonNode.fieldNames(); it.hasNext();) {
+ String key = it.next();
+ final Type s;
+ if (type != null && type.typeId().equals(Type.TypeID.MAP)) {
+ s = ((Types.MapType)type).valueType();
+ } else if (type != null && type.typeId().equals(Type.TypeID.RECORD)) {
+ s = ((Types.RecordType)type).field(key).type();
+ } else {
+ s = null;
+ }
+ Object value = toObject(jsonNode.get(key), s);
+ m.put(key, value);
+ }
+ return m;
+ }
+ return null;
+ }
+
+ public static String toJson(List<InternalSchema> internalSchemas) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ generator.writeStartObject();
+ generator.writeArrayFieldStart(SCHEMAS);
+ for (InternalSchema schema : internalSchemas) {
+ toJson(schema, generator);
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static String toJson(InternalSchema internalSchema) {
+ try {
+ StringWriter writer = new StringWriter();
+ JsonGenerator generator = (new JsonFactory()).createGenerator(writer);
+ toJson(internalSchema, generator);
+ generator.flush();
+ return writer.toString();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static void toJson(InternalSchema internalSchema, JsonGenerator generator) throws IOException {
+ toJson(internalSchema.getRecord(), internalSchema.getMax_column_id(), internalSchema.schemaId(), generator);
+ }
+
+ public static void toJson(Types.RecordType record, Integer maxColumnId, Long versionId, JsonGenerator generator) throws IOException {
+ generator.writeStartObject();
+ if (maxColumnId != null) {
+ generator.writeNumberField(MAX_COLUMN_ID, maxColumnId);
+ }
+ if (versionId != null) {
+ generator.writeNumberField(VERSION_ID, versionId);
+ }
+ generator.writeStringField(TYPE, RECORD);
+ generator.writeArrayFieldStart(FIELDS);
+ for (Types.Field field : record.fields()) {
+ generator.writeStartObject();
+ generator.writeNumberField(ID, field.fieldId());
+ generator.writeStringField(NAME, field.name());
+ generator.writeBooleanField(OPTIONAL, field.isOptional());
+ generator.writeFieldName(TYPE);
+ toJson(field.type(), generator);
+ if (field.doc() != null) {
+ generator.writeStringField(DOC, field.doc());
+ }
+ generator.writeEndObject();
+ }
+ generator.writeEndArray();
+ generator.writeEndObject();
+ }
+
+ static void toJson(Type type, JsonGenerator generator) throws IOException {
+ switch (type.typeId()) {
+ case RECORD:
+ toJson((Types.RecordType) type, null, null, generator);
+ break;
+ case ARRAY:
+ Types.ArrayType array = (Types.ArrayType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, ARRAY);
+ generator.writeNumberField(ELEMENT_ID, array.elementId());
+ generator.writeFieldName(ELEMENT);
+ toJson(array.elementType(), generator);
+ generator.writeBooleanField(ELEMENT_OPTIONAL, array.isElementOptional());
+ generator.writeEndObject();
+ break;
+ case MAP:
+ Types.MapType map = (Types.MapType) type;
+ generator.writeStartObject();
+ generator.writeStringField(TYPE, MAP);
+ generator.writeNumberField(KEY_ID, map.keyId());
+ generator.writeFieldName(KEY);
+ toJson(map.keyType(), generator);
+ generator.writeNumberField(VALUE_ID, map.valueId());
+ generator.writeFieldName(VALUE);
+ toJson(map.valueType(), generator);
+ generator.writeBooleanField(VALUE_OPTIONAL, map.isValueOptional());
+ generator.writeEndObject();
+ break;
+ default:
+ if (!type.isNestedType()) {
+ generator.writeString(type.toString());
+ } else {
+ throw new IllegalArgumentIOException(String.format("cannot write unknown types: %s", type));
+ }
+ }
+ }
+
+ private static Type parserTypeFromJson(JsonNode jsonNode) {
+ if (jsonNode.isTextual()) {
+ String type = jsonNode.asText().toLowerCase(Locale.ROOT);
+ // deal with fixed and decimal
+ Matcher fixed = FIXED.matcher(type);
+ if (fixed.matches()) {
+ return Types.FixedType.getFixed(Integer.parseInt(fixed.group(1)));
+ }
+ Matcher decimal = DECIMAL.matcher(type);
+ if (decimal.matches()) {
+ return Types.DecimalType.get(
+ Integer.parseInt(decimal.group(1)),
+ Integer.parseInt(decimal.group(2)));
+ }
+ // deal with other type
+ switch (Type.fromValue(type)) {
+ case BOOLEAN:
+ return Types.BooleanType.get();
+ case INT:
+ return Types.IntType.get();
+ case LONG:
+ return Types.LongType.get();
+ case FLOAT:
+ return Types.FloatType.get();
+ case DOUBLE:
+ return Types.DoubleType.get();
+ case DATE:
+ return Types.DateType.get();
+ case TIME:
+ return Types.TimeType.get();
+ case TIMESTAMP:
+ return Types.TimestampType.get();
+ case STRING:
+ return Types.StringType.get();
+ case UUID:
+ return Types.UUIDType.get();
+ case BINARY:
+ return Types.BinaryType.get();
+ default:
+ throw new IllegalArgumentException("cannot parser types from jsonNode");
+ }
+ } else if (jsonNode.isObject()) {
+ String typeStr = jsonNode.get(TYPE).asText();
+ if (RECORD.equals(typeStr)) {
+ JsonNode fieldNodes = jsonNode.get(FIELDS);
+ Iterator<JsonNode> iter = fieldNodes.elements();
+ List<Types.Field> fields = new ArrayList<>();
+ while (iter.hasNext()) {
+ JsonNode field = iter.next();
+ // extract
+ int id = field.get(ID).asInt();
+ String name = field.get(NAME).asText();
+ Type type = parserTypeFromJson(field.get(TYPE));
+ String doc = field.has(DOC) ? field.get(DOC).asText() : null;
+ boolean optional = field.get(OPTIONAL).asBoolean();
+ // build fields
+ fields.add(Types.Field.get(id, optional, name, type, doc));
+ }
+ return Types.RecordType.get(fields);
+ } else if (ARRAY.equals(typeStr)) {
+ int elementId = jsonNode.get(ELEMENT_ID).asInt();
+ Type elementType = parserTypeFromJson(jsonNode.get(ELEMENT));
+ boolean optional = jsonNode.get(ELEMENT_OPTIONAL).asBoolean();
+ return Types.ArrayType.get(elementId, optional, elementType);
+ } else if (MAP.equals(typeStr)) {
+ int keyId = jsonNode.get(KEY_ID).asInt();
+ Type keyType = parserTypeFromJson(jsonNode.get(KEY));
+ int valueId = jsonNode.get(VALUE_ID).asInt();
+ Type valueType = parserTypeFromJson(jsonNode.get(VALUE));
+ boolean optional = jsonNode.get(VALUE_OPTIONAL).asBoolean();
+ return Types.MapType.get(keyId, valueId, keyType, valueType, optional);
+ }
+ }
+ throw new IllegalArgumentException(String.format("cannot parse type from jsonNode: %s", jsonNode));
+ }
+
+ public static InternalSchema fromJson(JsonNode jsonNode) {
+ Integer maxColumnId = !jsonNode.has(MAX_COLUMN_ID) ? null : jsonNode.get(MAX_COLUMN_ID).asInt();
+ Long versionId = !jsonNode.has(VERSION_ID) ? null : jsonNode.get(VERSION_ID).asLong();
+ Types.RecordType type = (Types.RecordType)parserTypeFromJson(jsonNode);
+ if (versionId == null) {
+ return new InternalSchema(type.fields());
+ } else {
+ if (maxColumnId != null) {
+ return new InternalSchema(versionId, maxColumnId, type.fields());
+ } else {
+ return new InternalSchema(versionId, type.fields());
+ }
+ }
+ }
+
+ public static InternalSchema fromJson(String json) {
+ if (json == null || json.isEmpty()) {
+ return null;
+ }
+ try {
+ return fromJson((new ObjectMapper(new JsonFactory())).readValue(json, JsonNode.class));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static TreeMap<Long, InternalSchema> parserSchemas(String json) {
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #3668: [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#discussion_r709096594
##########
File path: hudi-common/pom.xml
##########
@@ -172,6 +172,12 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.github.ben-manes.caffeine</groupId>
Review comment:
what this dependency used for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-919855741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2235",
"triggerID" : "4b64c7ecf585993377b6a0d9b18e14003068471e",
"triggerType" : "PUSH"
}, {
"hash" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249",
"triggerID" : "101b85af0521d578a9a42eb50a8b523bc65d0abf",
"triggerType" : "PUSH"
}, {
"hash" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba407cc5af64d50eaa18f5df4adf1307853c264d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 101b85af0521d578a9a42eb50a8b523bc65d0abf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2249)
* ba407cc5af64d50eaa18f5df4adf1307853c264d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #3668: [RFC-33] [HUDI-2429][WIP] Full schema evolution
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #3668:
URL: https://github.com/apache/hudi/pull/3668#issuecomment-939993169
@xiarixiaoyao It would really help if you could share a gist showing the schema evolution steps. For example, earlier I tried [this](https://gist.github.com/codope/9ebb60d5763d998aa2f11914729263c6) and add/drop/reorder was working but rename was not working. Now, after the latest push, when I try the same but with the below spark-sql command, i don't see schema directory being created.
```
./bin/spark-sql --jars ${JARS} \
--packages org.apache.spark:spark-avro_2.11:2.4.7,com.github.ben-manes.caffeine:caffeine:2.9.1 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.kryoserializer.buffer.max=1024m' --conf "spark.memory.storageFraction=0.8" \
--conf "hoodie.schema.evolution.enable=true"
--conf spark.rdd.compress=true --driver-memory 2g --conf "spark.memory.fraction=0.8"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org