You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/15 17:15:39 UTC
[GitHub] [hudi] fengjian428 opened a new pull request #3671: Hudi 2418 add HiveSchemaProvider
fengjian428 opened a new pull request #3671:
URL: https://github.com/apache/hudi/pull/3671
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
https://issues.apache.org/jira/browse/HUDI-2418
--schemaprovider-class org.apache.hudi.utilities.schema.HiveSchemaProvider\
--hoodie-conf hoodie.deltastreamer.schemaprovider.source.schema.hive.database=database\
--hoodie-conf hoodie.deltastreamer.schemaprovider.source.schema.hive.table=source_table\
--hoodie-conf hoodie.deltastreamer.schemaprovider.target.schema.hive.database= database\
--hoodie-conf hoodie.deltastreamer.schemaprovider.target.schema.hive.table=target_table\
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963514430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717431276
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Can keep this ？ if it just a suggestion and this class only for unittest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r715707224
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,106 @@
+package org.apache.hudi.utilities.functional;
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717431276
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Can keep this ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716837947
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBandTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
Review comment:
> Can we use the pre-defined constant?
is this a must？ found other schema provider test class also wrote like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-921517144
@fengjian428 thanks for your contribution. Please check the CI failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716530955
##########
File path: hudi-utilities/src/test/resources/delta-streamer-config/hive_schema_provider_target.avsc
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-952137960
> @fengjian428 FYI. error from CI test. seems related to the change
>
> ```
> [ERROR] TestHiveSchemaProvider.testSourceSchema() Time elapsed: 0.569 s <<< ERROR!
> org.apache.spark.sql.AnalysisException:
> Hive support is required to CREATE Hive TABLE (AS SELECT);;
> 'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
>
>
> [ERROR] TestHiveSchemaProvider.testTargetSchema() Time elapsed: 0.024 s <<< ERROR!
> org.apache.spark.sql.AnalysisException:
> Hive support is required to CREATE Hive TABLE (AS SELECT);;
> 'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
>
>
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] Hive support is required to CREATE Hive TABLE (AS SELECT);;
> 'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
>
> [ERROR] Hive support is required to CREATE Hive TABLE (AS SELECT);;
> 'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
> ```
The latest CI is success
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b744966057d6850bbfa85b0d79750fb3ca3b0325 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475)
* 2593606faf5365b7a8f4816ad85572b4d888bd04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-961588424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2593606faf5365b7a8f4816ad85572b4d888bd04 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833)
* 48f27001b3fe89ab09314e986784eec74c202d65 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r738320495
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestHiveSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ );
+ for (Schema.Field field : sourceSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc"));
+ for (Schema.Field field : targetSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
Review comment:
If it is not expected, then throw it to let it fail fast?
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestHiveSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ );
+ for (Schema.Field field : sourceSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc"));
+ for (Schema.Field field : targetSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
+ }
+ }
+
+ @Test
+ public void testNotExistTable() {
+ String wrongName = "wrong_schema_tab";
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", wrongName);
+ Assertions.assertThrows(NoSuchTableException.class, () -> {
+ try {
+ UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ } catch (Throwable exception) {
+ while (exception.getCause() != null) {
+ exception = exception.getCause();
+ }
+ throw exception;
+ }
+ });
+ }
+
+ private static Pair<String, String> getDBAndTableName(String fullName) {
Review comment:
`getDBAndTableName ` -> `paresDBAndTableName` looks better?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ }
+ } catch (NoSuchTableException | NoSuchDatabaseException e) {
+ List<String> tables = new ArrayList<String>() {
+ {
+ add(sourceSchemaTableName);
+ }
+ };
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ tables.add(props.getString(Config.SOURCE_SCHEMA_TABLE_PROP));
+ }
+ String message = String.format("Can't find Hive table(s): %s", String.join(",", tables));
+ throw new IllegalArgumentException(message, e);
+ }
+ }
+
+ @Override
+ public Schema getSourceSchema() {
+ return sourceSchema;
+ }
+
+ @Override
+ public Schema getTargetSchema() {
+ if (targetSchema != null) {
+ return targetSchema;
+ } else {
+ return super.getTargetSchema();
+ }
+ }
+
+}
Review comment:
End with an `enter`?
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestHiveSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ );
+ for (Schema.Field field : sourceSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
Review comment:
ditto, do not eat the exception if it is not expected?
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestHiveSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ );
+ for (Schema.Field field : sourceSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ Schema originalSchema = new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc"));
+ for (Schema.Field field : targetSchema.getFields()) {
+ Schema.Field originalField = originalSchema.getField(field.name());
+ assertTrue(originalField != null);
+ }
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
+ }
+ }
+
+ @Test
+ public void testNotExistTable() {
+ String wrongName = "wrong_schema_tab";
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", wrongName);
+ Assertions.assertThrows(NoSuchTableException.class, () -> {
+ try {
+ UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ } catch (Throwable exception) {
+ while (exception.getCause() != null) {
+ exception = exception.getCause();
+ }
+ throw exception;
+ }
+ });
+ }
+
+ private static Pair<String, String> getDBAndTableName(String fullName) {
+ String[] dbAndTableName = fullName.split("\\.");
+ if (dbAndTableName.length > 1) {
+ return new ImmutablePair<>(dbAndTableName[0], dbAndTableName[1]);
+ } else {
+ return new ImmutablePair<>("default", dbAndTableName[0]);
+ }
+ }
+
+ private void createSchemaTable(String fullName) throws IOException {
+ SparkSession spark = spark();
+ String createTableSQL = UtilitiesTestBase.Helpers.readFile(String.format("delta-streamer-config/%s.sql", fullName));
+ Pair<String, String> dbAndTableName = getDBAndTableName(fullName);
+ spark.sql(String.format("CREATE DATABASE IF NOT EXISTS %s", dbAndTableName.getLeft()));
+ spark.sql(createTableSQL);
+ spark.sql(String.format("SHOW CREATE TABLE %s.%s", dbAndTableName.getLeft(), dbAndTableName.getRight())).show(false);
Review comment:
Is it necessary to run the `show create table`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0312ca5d38be2a3385d4910a70b6055e85faa1cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281)
* 8196d217ba88ff5b9cd7ce02cbb68539a007abf2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c77bd85c92bc324546c39a8d02fd206ba148d9ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717386107
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBandTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Just a suggestion not MUST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-921991089
> @fengjian428 thanks for your contribution. Please check the CI failure.
seems has check style error, already fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717386107
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBandTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Just a suggestion not MUST.
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
ditto
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
Review comment:
ditto
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ assertEquals(
+ targetSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
+ }
+ }
+
+ @Test
+ public void testNotExistTable() {
+ String wrongName = "wrong_schema_tab";
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", wrongName);
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r741649195
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -164,7 +164,7 @@ public synchronized void runBeforeEach() {
SparkConf sparkConf = conf();
SparkRDDWriteClient.registerClasses(sparkConf);
HoodieReadClient.addHoodieSupport(sparkConf);
- spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ spark = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
Review comment:
instead of changing this for all tests, can you override `org.apache.hudi.testutils.providers.SparkProvider#conf()` in your specific test class to pass in the configs you need for your testcase?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ }
+ } catch (NoSuchTableException | NoSuchDatabaseException e) {
+ List<String> tables = new ArrayList<String>() {
+ {
+ add(sourceSchemaTableName);
+ }
+ };
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ tables.add(props.getString(Config.SOURCE_SCHEMA_TABLE_PROP));
+ }
+ String message = String.format("Can't find Hive table(s): %s", String.join(",", tables));
Review comment:
these lines look weird and verbose.. you just need to log 2 variables i guess? why not just format them all at once in one line? also i think you're trying to `props.getString(Config.TARGET_SCHEMA_TABLE_PROP)` instead.
catch block should be lightweight on logic. you should be able to `props.get()` with a default `""` to help formating to save the if checking. Can you simplify these lines pls
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
Review comment:
have you tested this with different spark versions? 2.4, 3.0.1, 3.1, etc. we want to thorough about this.
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -164,7 +164,7 @@ public synchronized void runBeforeEach() {
SparkConf sparkConf = conf();
SparkRDDWriteClient.registerClasses(sparkConf);
HoodieReadClient.addHoodieSupport(sparkConf);
- spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ spark = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
Review comment:
instead of changing this for all tests, can you override `org.apache.hudi.testutils.providers.SparkProvider#conf()` in your specific test class to pass in the configs you need for your testcase?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ }
+ } catch (NoSuchTableException | NoSuchDatabaseException e) {
+ List<String> tables = new ArrayList<String>() {
+ {
+ add(sourceSchemaTableName);
+ }
+ };
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ tables.add(props.getString(Config.SOURCE_SCHEMA_TABLE_PROP));
+ }
+ String message = String.format("Can't find Hive table(s): %s", String.join(",", tables));
Review comment:
these lines look weird and verbose.. you just need to log 2 variables i guess? why not just format them all at once in one line? also i think you're trying to `props.getString(Config.TARGET_SCHEMA_TABLE_PROP)` instead.
catch block should be lightweight on logic. you should be able to `props.get()` with a default `""` to help formating to save the if checking. Can you simplify these lines pls
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
Review comment:
have you tested this with different spark versions? 2.4, 3.0.1, 3.1, etc. we want to thorough about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-961588424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48f27001b3fe89ab09314e986784eec74c202d65 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864)
* c76d2c290f10e235305a2b782519e20845e328cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96258479d15400d35f418d20d2fcd1301e604f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940)
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96258479d15400d35f418d20d2fcd1301e604f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f95bba83e08949ef6b3f986409768978752b08ad Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2593606faf5365b7a8f4816ad85572b4d888bd04 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r715706708
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,91 @@
+package org.apache.hudi.utilities.schema;
Review comment:
Firstly, you must add the Apache License header on top of generic files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8196d217ba88ff5b9cd7ce02cbb68539a007abf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399)
* c305e93d3e7e8028f6fd2bc78be6e2c04e85e184 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716239631
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.FunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends FunctionalTestHarness {
Review comment:
@fengjian428 can you rebase on master please? `FunctionalTestHarness` has been deprecated in favor of `SparkClientFunctionalTestHarness`. please follow the deprecation notes. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c305e93d3e7e8028f6fd2bc78be6e2c04e85e184 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404)
* 899581b6a845086105ee586812380b7fd0e006e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963433091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963486231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963426386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-961588424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0312ca5d38be2a3385d4910a70b6055e85faa1cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-929787041
@fengjian428 Check CI again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-961588424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963424292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963424292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b744966057d6850bbfa85b0d79750fb3ca3b0325 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475)
* 2593606faf5365b7a8f4816ad85572b4d888bd04 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-984318178
sorry for the late reply. Will have a final check soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f95bba83e08949ef6b3f986409768978752b08ad Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917)
* 96258479d15400d35f418d20d2fcd1301e604f5c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716484364
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ LOG.warn(this.targetSchema.toString(true));
Review comment:
Why `warn`? And this is a bad log practice. You should add more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0312ca5d38be2a3385d4910a70b6055e85faa1cd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281)
* 8196d217ba88ff5b9cd7ce02cbb68539a007abf2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-927286542
@fengjian428 Thanks for addressing my concerns. But, the Azure CI still failed. Please check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c77bd85c92bc324546c39a8d02fd206ba148d9ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242)
* 0312ca5d38be2a3385d4910a70b6055e85faa1cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2593606faf5365b7a8f4816ad85572b4d888bd04 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833)
* 48f27001b3fe89ab09314e986784eec74c202d65 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35e7f483618be007802784caa5c64b953cc0ffda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436)
* b744966057d6850bbfa85b0d79750fb3ca3b0325 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-929816532
> @fengjian428 Check CI again?
what you mean? the checks below all passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963865781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
}, {
"hash" : "eafa89d286b157d626d93704d8d549801b41ba68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eafa89d286b157d626d93704d8d549801b41ba68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
* eafa89d286b157d626d93704d8d549801b41ba68 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963433091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963486231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963866868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
}, {
"hash" : "eafa89d286b157d626d93704d8d549801b41ba68",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239",
"triggerID" : "eafa89d286b157d626d93704d8d549801b41ba68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
* eafa89d286b157d626d93704d8d549801b41ba68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c305e93d3e7e8028f6fd2bc78be6e2c04e85e184 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716530805
##########
File path: hudi-utilities/src/test/resources/delta-streamer-config/hive_schema_provider_source.avsc
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
Review comment:
add an empty line to separate content with the license header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717387264
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-938603036
@fengjian428 Would you please recheck the azure CI? Seems still failing..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c77bd85c92bc324546c39a8d02fd206ba148d9ca Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242)
* 0312ca5d38be2a3385d4910a70b6055e85faa1cd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716492353
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ LOG.warn(this.targetSchema.toString(true));
Review comment:
sorry this is only for test, already removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c7664dd776e18d31f46db5eff91dd3d87bf590c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426)
* 35e7f483618be007802784caa5c64b953cc0ffda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35e7f483618be007802784caa5c64b953cc0ffda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717387781
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ assertEquals(
+ targetSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
+ }
+ }
+
+ @Test
+ public void testNotExistTable() {
+ String wrongName = "wrong_schema_tab";
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", wrongName);
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717387108
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-954451046
@xushiyan Any input?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963428769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963890670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
}, {
"hash" : "eafa89d286b157d626d93704d8d549801b41ba68",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239",
"triggerID" : "eafa89d286b157d626d93704d8d549801b41ba68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eafa89d286b157d626d93704d8d549801b41ba68 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 96258479d15400d35f418d20d2fcd1301e604f5c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940)
* f21e9f352a40b284926a1007363d68a0d3fca0db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c76d2c290f10e235305a2b782519e20845e328cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48f27001b3fe89ab09314e986784eec74c202d65 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864)
* c76d2c290f10e235305a2b782519e20845e328cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r741649195
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -164,7 +164,7 @@ public synchronized void runBeforeEach() {
SparkConf sparkConf = conf();
SparkRDDWriteClient.registerClasses(sparkConf);
HoodieReadClient.addHoodieSupport(sparkConf);
- spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ spark = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
Review comment:
instead of changing this for all tests, can you override `org.apache.hudi.testutils.providers.SparkProvider#conf()` in your specific test class to pass in the configs you need for your testcase?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ }
+ } catch (NoSuchTableException | NoSuchDatabaseException e) {
+ List<String> tables = new ArrayList<String>() {
+ {
+ add(sourceSchemaTableName);
+ }
+ };
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ tables.add(props.getString(Config.SOURCE_SCHEMA_TABLE_PROP));
+ }
+ String message = String.format("Can't find Hive table(s): %s", String.join(",", tables));
Review comment:
these lines look weird and verbose.. you just need to log 2 variables i guess? why not just format them all at once in one line? also i think you're trying to `props.getString(Config.TARGET_SCHEMA_TABLE_PROP)` instead.
catch block should be lightweight on logic. you should be able to `props.get()` with a default `""` to help formating to save the if checking. Can you simplify these lines pls
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
Review comment:
have you tested this with different spark versions? 2.4, 3.0.1, 3.1, etc. we want to thorough about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r744944449
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -164,7 +164,7 @@ public synchronized void runBeforeEach() {
SparkConf sparkConf = conf();
SparkRDDWriteClient.registerClasses(sparkConf);
HoodieReadClient.addHoodieSupport(sparkConf);
- spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ spark = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
Review comment:
In enableHiveSupport function, spark also check if hiveClassesArePresent, if not then throw a exception, do you mean I should pass config directly and ignore this logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963426386
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f21e9f352a40b284926a1007363d68a0d3fca0db Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946)
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963514430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963865781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
}, {
"hash" : "eafa89d286b157d626d93704d8d549801b41ba68",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eafa89d286b157d626d93704d8d549801b41ba68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
* eafa89d286b157d626d93704d8d549801b41ba68 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r745485740
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
Review comment:
I was using HiveSchemaProvider in 2.4 for long time, today I tried spark 3.0.1, also work fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35e7f483618be007802784caa5c64b953cc0ffda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436)
* b744966057d6850bbfa85b0d79750fb3ca3b0325 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8196d217ba88ff5b9cd7ce02cbb68539a007abf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r717431276
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Can keep this ？
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBAndTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBAndTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Can keep this ？ if it just a suggestion and this class only for unittest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-930287792
> > @fengjian428 Check CI again?
>
> what you mean? the checks below all passed
I mean the Azure CI, please check [here](https://github.com/apache/hudi/pull/3671#issuecomment-920215882).
Now, Hudi's CI contains two parts.
Part 1: The github workflows;
Part 2: The Azure CI;
The merge condition is that the two parts are both successful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-929816532
> @fengjian428 Check CI again?
what you mean? the checks below all passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8196d217ba88ff5b9cd7ce02cbb68539a007abf2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399)
* c305e93d3e7e8028f6fd2bc78be6e2c04e85e184 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c7664dd776e18d31f46db5eff91dd3d87bf590c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426)
* 35e7f483618be007802784caa5c64b953cc0ffda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-929787041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716531592
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBandTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", dbAndTableName.getRight());
+ }
+
+ @Test
+ public void testSourceSchema() throws Exception {
+ try {
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ Schema sourceSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ assertEquals(
+ sourceSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_source.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source schema. ", e);
+ }
+ }
+
+ @Test
+ public void testTargetSchema() throws Exception {
+ try {
+ Pair<String, String> dbAndTableName = getDBandTableName(TARGET_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.database", dbAndTableName.getLeft());
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.target.schema.hive.table", dbAndTableName.getRight());
+ createSchemaTable(SOURCE_SCHEMA_TABLE_NAME);
+ createSchemaTable(TARGET_SCHEMA_TABLE_NAME);
+ Schema targetSchema = UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getTargetSchema();
+ assertEquals(
+ targetSchema.toString().toUpperCase(),
+ new Schema.Parser().parse(
+ UtilitiesTestBase.Helpers.readFile("delta-streamer-config/hive_schema_provider_target.avsc")
+ ).toString().toUpperCase());
+ } catch (HoodieException e) {
+ LOG.error("Failed to get source/target schema. ", e);
+ }
+ }
+
+ @Test
+ public void testNotExistTable() {
+ String wrongName = "wrong_schema_tab";
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.table", wrongName);
+ Assertions.assertThrows(NoSuchTableException.class, () -> {
+ try {
+ UtilHelpers.createSchemaProvider(HiveSchemaProvider.class.getName(), PROPS, jsc()).getSourceSchema();
+ } catch (Throwable exception) {
+ while (exception.getCause() != null) {
+ exception = exception.getCause();
+ }
+ throw exception;
+ }
+ });
+ }
+
+ private static Pair<String, String> getDBandTableName(String fullName) {
Review comment:
`and` -> `And`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-931957889
> > > @fengjian428 Check CI again?
> >
> >
> > what you mean? the checks below all passed
>
> I mean the Azure CI, please check [here](https://github.com/apache/hudi/pull/3671#issuecomment-920215882).
>
> Now, Hudi's CI contains two parts.
>
> Part 1: The github workflows; Part 2: The Azure CI;
>
> The merge condition is that the two parts are both successful.
ok , added the missed dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c77bd85c92bc324546c39a8d02fd206ba148d9ca Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-978874060
@xushiyan @yanghua @vinothchandar any input?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-950183648
@fengjian428 FYI. error from CI test. seems related to the change
```
[ERROR] TestHiveSchemaProvider.testSourceSchema() Time elapsed: 0.569 s <<< ERROR!
org.apache.spark.sql.AnalysisException:
Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
[ERROR] TestHiveSchemaProvider.testTargetSchema() Time elapsed: 0.024 s <<< ERROR!
org.apache.spark.sql.AnalysisException:
Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
[ERROR] Hive support is required to CREATE Hive TABLE (AS SELECT);;
'CreateTable `schema_registry`.`source_schema_tab`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Ignore
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r741649195
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/SparkClientFunctionalTestHarness.java
##########
@@ -164,7 +164,7 @@ public synchronized void runBeforeEach() {
SparkConf sparkConf = conf();
SparkRDDWriteClient.registerClasses(sparkConf);
HoodieReadClient.addHoodieSupport(sparkConf);
- spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ spark = SparkSession.builder().config(sparkConf).enableHiveSupport().getOrCreate();
Review comment:
instead of changing this for all tests, can you override `org.apache.hudi.testutils.providers.SparkProvider#conf()` in your specific test class to pass in the configs you need for your testcase?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
+
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ String targetSchemaDBName = props.getString(Config.TARGET_SCHEMA_DATABASE_PROP, "default");
+ String targetSchemaTableName = props.getString(Config.TARGET_SCHEMA_TABLE_PROP);
+ TableIdentifier targetSchemaTable = new TableIdentifier(targetSchemaTableName, scala.Option.apply(targetSchemaDBName));
+ StructType targetSchema = spark.sessionState().catalog().getTableMetadata(targetSchemaTable).schema();
+ this.targetSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ targetSchema,
+ targetSchemaTableName,
+ "hoodie." + targetSchemaDBName);
+ }
+ } catch (NoSuchTableException | NoSuchDatabaseException e) {
+ List<String> tables = new ArrayList<String>() {
+ {
+ add(sourceSchemaTableName);
+ }
+ };
+ if (props.containsKey(Config.TARGET_SCHEMA_TABLE_PROP)) {
+ tables.add(props.getString(Config.SOURCE_SCHEMA_TABLE_PROP));
+ }
+ String message = String.format("Can't find Hive table(s): %s", String.join(",", tables));
Review comment:
these lines look weird and verbose.. you just need to log 2 variables i guess? why not just format them all at once in one line? also i think you're trying to `props.getString(Config.TARGET_SCHEMA_TABLE_PROP)` instead.
catch block should be lightweight on logic. you should be able to `props.get()` with a default `""` to help formating to save the if checking. Can you simplify these lines pls
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/HiveSchemaProvider.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.schema;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.AvroConversionUtils;
+import org.apache.hudi.DataSourceUtils;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.TableIdentifier;
+import org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.apache.spark.sql.types.StructType;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+public class HiveSchemaProvider extends SchemaProvider {
+
+ /**
+ * Configs supported.
+ */
+ public static class Config {
+ private static final String SOURCE_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.database";
+ private static final String SOURCE_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.source.schema.hive.table";
+ private static final String TARGET_SCHEMA_DATABASE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.database";
+ private static final String TARGET_SCHEMA_TABLE_PROP = "hoodie.deltastreamer.schemaprovider.target.schema.hive.table";
+ }
+
+ private static final Logger LOG = LogManager.getLogger(HiveSchemaProvider.class);
+
+ private final Schema sourceSchema;
+
+ private Schema targetSchema;
+
+ public HiveSchemaProvider(TypedProperties props, JavaSparkContext jssc) {
+ super(props, jssc);
+ DataSourceUtils.checkRequiredProperties(props, Collections.singletonList(Config.SOURCE_SCHEMA_TABLE_PROP));
+ String sourceSchemaDBName = props.getString(Config.SOURCE_SCHEMA_DATABASE_PROP, "default");
+ String sourceSchemaTableName = props.getString(Config.SOURCE_SCHEMA_TABLE_PROP);
+ SparkSession spark = SparkSession.builder().config(jssc.getConf()).enableHiveSupport().getOrCreate();
+ try {
+ TableIdentifier sourceSchemaTable = new TableIdentifier(sourceSchemaTableName, scala.Option.apply(sourceSchemaDBName));
+ StructType sourceSchema = spark.sessionState().catalog().getTableMetadata(sourceSchemaTable).schema();
+
+ this.sourceSchema = AvroConversionUtils.convertStructTypeToAvroSchema(
+ sourceSchema,
+ sourceSchemaTableName,
+ "hoodie." + sourceSchemaDBName);
Review comment:
have you tested this with different spark versions? 2.4, 3.0.1, 3.1, etc. we want to thorough about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #3671: [HUDI-2418] Support HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
xushiyan merged pull request #3671:
URL: https://github.com/apache/hudi/pull/3671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963866868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
}, {
"hash" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224",
"triggerID" : "2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e",
"triggerType" : "PUSH"
}, {
"hash" : "eafa89d286b157d626d93704d8d549801b41ba68",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239",
"triggerID" : "eafa89d286b157d626d93704d8d549801b41ba68",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceab6429c3f1fec97fe5af18ca2494f5dcc3e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3224)
* eafa89d286b157d626d93704d8d549801b41ba68 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f95bba83e08949ef6b3f986409768978752b08ad Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917)
* 96258479d15400d35f418d20d2fcd1301e604f5c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c76d2c290f10e235305a2b782519e20845e328cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904)
* f95bba83e08949ef6b3f986409768978752b08ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c7664dd776e18d31f46db5eff91dd3d87bf590c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716448372
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.FunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends FunctionalTestHarness {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c305e93d3e7e8028f6fd2bc78be6e2c04e85e184 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404)
* 899581b6a845086105ee586812380b7fd0e006e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 899581b6a845086105ee586812380b7fd0e006e9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425)
* 5c7664dd776e18d31f46db5eff91dd3d87bf590c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 899581b6a845086105ee586812380b7fd0e006e9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425)
* 5c7664dd776e18d31f46db5eff91dd3d87bf590c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#discussion_r716532569
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHiveSchemaProvider.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.functional;
+
+import org.apache.avro.Schema;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.ImmutablePair;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.SparkClientFunctionalTestHarness;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.schema.HiveSchemaProvider;
+import org.apache.hudi.utilities.testutils.UtilitiesTestBase;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@Tag("functional")
+public class TestHiveSchemaProvider extends SparkClientFunctionalTestHarness {
+ private static final Logger LOG = LogManager.getLogger(TestJdbcbasedSchemaProvider.class);
+ private static final TypedProperties PROPS = new TypedProperties();
+ private static final String SOURCE_SCHEMA_TABLE_NAME = "schema_registry.source_schema_tab";
+ private static final String TARGET_SCHEMA_TABLE_NAME = "schema_registry.target_schema_tab";
+
+ @BeforeAll
+ public static void init() {
+ Pair<String, String> dbAndTableName = getDBandTableName(SOURCE_SCHEMA_TABLE_NAME);
+ PROPS.setProperty("hoodie.deltastreamer.schemaprovider.source.schema.hive.database", dbAndTableName.getLeft());
Review comment:
Can we use the pre-defined constant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [Hudi-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c77bd85c92bc324546c39a8d02fd206ba148d9ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #3671: [HUDI-2418] Support HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-985236883
@xushiyan feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48f27001b3fe89ab09314e986784eec74c202d65 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c76d2c290f10e235305a2b782519e20845e328cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904)
* f95bba83e08949ef6b3f986409768978752b08ad Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-925794836
this is a pretty interesting change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-920215882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b744966057d6850bbfa85b0d79750fb3ca3b0325 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3671: [HUDI-2418] add HiveSchemaProvider
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3671:
URL: https://github.com/apache/hudi/pull/3671#issuecomment-963428769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2242",
"triggerID" : "c77bd85c92bc324546c39a8d02fd206ba148d9ca",
"triggerType" : "PUSH"
}, {
"hash" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2281",
"triggerID" : "0312ca5d38be2a3385d4910a70b6055e85faa1cd",
"triggerType" : "PUSH"
}, {
"hash" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2399",
"triggerID" : "8196d217ba88ff5b9cd7ce02cbb68539a007abf2",
"triggerType" : "PUSH"
}, {
"hash" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2404",
"triggerID" : "c305e93d3e7e8028f6fd2bc78be6e2c04e85e184",
"triggerType" : "PUSH"
}, {
"hash" : "899581b6a845086105ee586812380b7fd0e006e9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2425",
"triggerID" : "899581b6a845086105ee586812380b7fd0e006e9",
"triggerType" : "PUSH"
}, {
"hash" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2426",
"triggerID" : "5c7664dd776e18d31f46db5eff91dd3d87bf590c",
"triggerType" : "PUSH"
}, {
"hash" : "35e7f483618be007802784caa5c64b953cc0ffda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2436",
"triggerID" : "35e7f483618be007802784caa5c64b953cc0ffda",
"triggerType" : "PUSH"
}, {
"hash" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2475",
"triggerID" : "b744966057d6850bbfa85b0d79750fb3ca3b0325",
"triggerType" : "PUSH"
}, {
"hash" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2833",
"triggerID" : "2593606faf5365b7a8f4816ad85572b4d888bd04",
"triggerType" : "PUSH"
}, {
"hash" : "48f27001b3fe89ab09314e986784eec74c202d65",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2864",
"triggerID" : "48f27001b3fe89ab09314e986784eec74c202d65",
"triggerType" : "PUSH"
}, {
"hash" : "c76d2c290f10e235305a2b782519e20845e328cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2904",
"triggerID" : "c76d2c290f10e235305a2b782519e20845e328cc",
"triggerType" : "PUSH"
}, {
"hash" : "f95bba83e08949ef6b3f986409768978752b08ad",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2917",
"triggerID" : "f95bba83e08949ef6b3f986409768978752b08ad",
"triggerType" : "PUSH"
}, {
"hash" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2940",
"triggerID" : "96258479d15400d35f418d20d2fcd1301e604f5c",
"triggerType" : "PUSH"
}, {
"hash" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2946",
"triggerID" : "f21e9f352a40b284926a1007363d68a0d3fca0db",
"triggerType" : "PUSH"
}, {
"hash" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223",
"triggerID" : "2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f7fb95d2e17c4ff9aa6b2ea6b464f8ae9198bee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org