You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/06/30 04:51:31 UTC
[GitHub] [hudi] hddong opened a new pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
hddong opened a new pull request #1774:
URL: https://github.com/apache/hudi/pull/1774
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-657907476
> @yanghua please have a review when free.
OK, will review soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-657314447
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1774?src=pr&el=h1) Report
> Merging [#1774](https://codecov.io/gh/apache/hudi/pull/1774?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/8919be6a5d8038db7265bfd7459d72fbd545f133&el=desc) will **decrease** coverage by `2.26%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/1774?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1774 +/- ##
============================================
- Coverage 62.82% 60.55% -2.27%
- Complexity 3437 3646 +209
============================================
Files 401 440 +39
Lines 17091 19059 +1968
Branches 1698 1923 +225
============================================
+ Hits 10737 11541 +804
- Misses 5623 6730 +1107
- Partials 731 788 +57
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| #hudicli | `68.31% <0.00%> (?)` | `1429.00 <0.00> (?)` | |
| #hudiclient | `79.19% <ø> (-0.06%)` | `1257.00 <ø> (-1.00)` | |
| #hudicommon | `54.29% <ø> (+0.04%)` | `1486.00 <ø> (+1.00)` | |
| #hudihadoopmr | `38.96% <ø> (-0.40%)` | `163.00 <ø> (ø)` | |
| #hudihivesync | `72.25% <ø> (ø)` | `121.00 <ø> (ø)` | |
| #hudispark | `49.36% <ø> (+5.13%)` | `121.00 <ø> (+45.00)` | |
| #huditimelineservice | `63.47% <ø> (ø)` | `47.00 <ø> (ø)` | |
| #hudiutilities | `74.71% <ø> (+0.95%)` | `279.00 <ø> (-8.00)` | :arrow_up: |
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1774?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...rg/apache/hudi/cli/commands/HoodieSyncCommand.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0hvb2RpZVN5bmNDb21tYW5kLmphdmE=) | `2.85% <0.00%> (ø)` | `1.00 <0.00> (?)` | |
| [...ain/java/org/apache/hudi/cli/utils/CommitUtil.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL3V0aWxzL0NvbW1pdFV0aWwuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0Fic3RyYWN0SG9vZGllV3JpdGVDbGllbnQuamF2YQ==) | `74.07% <0.00%> (-4.23%)` | `18.00% <0.00%> (-3.00%)` | |
| [...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL0hvb2RpZVJPVGFibGVQYXRoRmlsdGVyLmphdmE=) | `65.71% <0.00%> (-2.95%)` | `12.00% <0.00%> (ø%)` | |
| [.../table/action/commit/BaseCommitActionExecutor.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdGFibGUvYWN0aW9uL2NvbW1pdC9CYXNlQ29tbWl0QWN0aW9uRXhlY3V0b3IuamF2YQ==) | `84.25% <0.00%> (-0.70%)` | `23.00% <0.00%> (-1.00%)` | |
| [...apache/hudi/keygen/NonpartitionedKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1zcGFyay9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9rZXlnZW4vTm9ucGFydGl0aW9uZWRLZXlHZW5lcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...hudi/hadoop/hive/HoodieCombineHiveInputFormat.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1oYWRvb3AtbXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaGFkb29wL2hpdmUvSG9vZGllQ29tYmluZUhpdmVJbnB1dEZvcm1hdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (ø%)` | |
| [...lities/exception/HoodieDeltaStreamerException.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2V4Y2VwdGlvbi9Ib29kaWVEZWx0YVN0cmVhbWVyRXhjZXB0aW9uLmphdmE=) | | | |
| [...i/utilities/keygen/TimestampBasedKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2tleWdlbi9UaW1lc3RhbXBCYXNlZEtleUdlbmVyYXRvci5qYXZh) | | | |
| [...pache/hudi/cli/commands/FileSystemViewCommand.java](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree#diff-aHVkaS1jbGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpL2NvbW1hbmRzL0ZpbGVTeXN0ZW1WaWV3Q29tbWFuZC5qYXZh) | `82.60% <0.00%> (ø)` | `19.00% <0.00%> (?%)` | |
| ... and [49 more](https://codecov.io/gh/apache/hudi/pull/1774/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1774:
URL: https://github.com/apache/hudi/pull/1774
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#discussion_r459376057
##########
File path: hudi-spark/src/test/java/HoodieJavaGenerateApp.java
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.hive.MultiPartKeysValueExtractor;
+import org.apache.hudi.hive.NonPartitionedExtractor;
+import org.apache.hudi.keygen.NonpartitionedKeyGenerator;
+import org.apache.hudi.keygen.SimpleKeyGenerator;
+import org.apache.hudi.testutils.DataSourceTestUtils;
+import org.apache.hudi.testutils.HoodieTestDataGenerator;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HoodieJavaGenerateApp {
+ @Parameter(names = {"--table-path", "-p"}, description = "path for Hoodie sample table")
+ private String tablePath = "file:///tmp/hoodie/sample-table";
+
+ @Parameter(names = {"--table-name", "-n"}, description = "table name for Hoodie sample table")
+ private String tableName = "hoodie_test";
+
+ @Parameter(names = {"--table-type", "-t"}, description = "One of COPY_ON_WRITE or MERGE_ON_READ")
+ private String tableType = HoodieTableType.COPY_ON_WRITE.name();
+
+ @Parameter(names = {"--hive-sync", "-hv"}, description = "Enable syncing to hive")
Review comment:
Why it is not `-hs`? Does it cause conflicts?
##########
File path: hudi-spark/src/test/java/HoodieJavaGenerateApp.java
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.hive.MultiPartKeysValueExtractor;
+import org.apache.hudi.hive.NonPartitionedExtractor;
+import org.apache.hudi.keygen.NonpartitionedKeyGenerator;
+import org.apache.hudi.keygen.SimpleKeyGenerator;
+import org.apache.hudi.testutils.DataSourceTestUtils;
+import org.apache.hudi.testutils.HoodieTestDataGenerator;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HoodieJavaGenerateApp {
+ @Parameter(names = {"--table-path", "-p"}, description = "path for Hoodie sample table")
+ private String tablePath = "file:///tmp/hoodie/sample-table";
+
+ @Parameter(names = {"--table-name", "-n"}, description = "table name for Hoodie sample table")
+ private String tableName = "hoodie_test";
+
+ @Parameter(names = {"--table-type", "-t"}, description = "One of COPY_ON_WRITE or MERGE_ON_READ")
+ private String tableType = HoodieTableType.COPY_ON_WRITE.name();
+
+ @Parameter(names = {"--hive-sync", "-hv"}, description = "Enable syncing to hive")
+ private Boolean enableHiveSync = false;
+
+ @Parameter(names = {"--hive-db", "-hd"}, description = "hive database")
+ private String hiveDB = "default";
+
+ @Parameter(names = {"--hive-table", "-ht"}, description = "hive table")
+ private String hiveTable = "hoodie_sample_test";
+
+ @Parameter(names = {"--hive-user", "-hu"}, description = "hive username")
Review comment:
Let's apply a unified rule about the first latter of the `description`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hddong commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
hddong commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-662496685
@yanghua : Thanks for your review, had address them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hddong commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
hddong commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-657322002
@yanghua please have a review when free.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#discussion_r457278729
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/HoodieSyncCommand.java
##########
@@ -74,9 +74,9 @@ public String validateSync(
}
String targetLatestCommit =
- targetTimeline.getInstants().iterator().hasNext() ? "0" : targetTimeline.lastInstant().get().getTimestamp();
+ targetTimeline.getInstants().iterator().hasNext() ? targetTimeline.lastInstant().get().getTimestamp() : "0";
Review comment:
Good catch!
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ Pair<String, String> stdOutErr;
+ if ("overwrite".equals(commitType)) {
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ // Set sleep time by hoodie.hiveserver.time.wait
+ Thread.sleep(getTimeWait());
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ stdOutErr = executeHiveCommand("show tables like '" + hiveTableName + "'");
+ assertTrue(stdOutErr.getLeft().isEmpty(), "Dropped table " + hiveTableName + " exists!");
+ }
+
+ // Run Hoodie Java App
+ String cmd = String.format("%s %s --hive-sync --table-path %s --hive-url %s --table-type %s --hive-table %s" +
+ " --commit-type %s --table-name %s", HOODIE_JAVA_APP, "HoodieJavaGenerateApp", hdfsUrl, HIVE_SERVER_JDBC_URL,
+ tableType, hiveTableName, commitType, hoodieTableName);
+ if (partitionType == PartitionType.MULTI_KEYS_PARTITIONED) {
+ cmd = cmd + " --use-multi-partition-keys";
+ } else if (partitionType == PartitionType.NON_PARTITIONED){
+ cmd = cmd + " --non-partitioned";
+ }
+ executeCommandStringInDocker(ADHOC_1_CONTAINER, cmd, true);
+
+ String snapshotTableName = getSnapshotTableName(tableType, hiveTableName);
+
+ // Ensure table does exist
+ stdOutErr = executeHiveCommand("show tables like '" + snapshotTableName + "'");
+ assertEquals(snapshotTableName, stdOutErr.getLeft(), "Table exists");
+ }
+
+ protected void dropHiveTables(String hiveTableName, String tableType) throws Exception {
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ.name())) {
+ executeHiveCommand("drop table if exists " + hiveTableName + "_rt");
+ executeHiveCommand("drop table if exists " + hiveTableName + "_ro");
+ } else {
+ executeHiveCommand("drop table if exists " + hiveTableName);
+ }
+ }
+
+ protected String getHdfsPath(String hiveTableName) {
Review comment:
`getHDFSPath` looks better?
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ Pair<String, String> stdOutErr;
+ if ("overwrite".equals(commitType)) {
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ // Set sleep time by hoodie.hiveserver.time.wait
+ Thread.sleep(getTimeWait());
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ stdOutErr = executeHiveCommand("show tables like '" + hiveTableName + "'");
+ assertTrue(stdOutErr.getLeft().isEmpty(), "Dropped table " + hiveTableName + " exists!");
Review comment:
I would not suggest using assertation in normal methods. We can throw an exception directly. WDYT?
##########
File path: hudi-spark/src/test/java/HoodieJavaGenerateApp.java
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.HoodieDataSourceHelpers;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.hive.MultiPartKeysValueExtractor;
+import org.apache.hudi.hive.NonPartitionedExtractor;
+import org.apache.hudi.keygen.NonpartitionedKeyGenerator;
+import org.apache.hudi.keygen.SimpleKeyGenerator;
+import org.apache.hudi.testutils.DataSourceTestUtils;
+import org.apache.hudi.testutils.HoodieTestDataGenerator;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.DataFrameWriter;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HoodieJavaGenerateApp {
+ @Parameter(names = {"--table-path", "-p"}, description = "path for Hoodie sample table")
+ private String tablePath = "file:///tmp/hoodie/sample-table";
+
+ @Parameter(names = {"--table-name", "-n"}, description = "table name for Hoodie sample table")
+ private String tableName = "hoodie_test";
+
+ @Parameter(names = {"--table-type", "-t"}, description = "One of COPY_ON_WRITE or MERGE_ON_READ")
+ private String tableType = HoodieTableType.COPY_ON_WRITE.name();
+
+ @Parameter(names = {"--hive-sync", "-hv"}, description = "Enable syncing to hive")
+ private Boolean enableHiveSync = false;
+
+ @Parameter(names = {"--hive-db", "-hd"}, description = "hive database")
+ private String hiveDB = "default";
+
+ @Parameter(names = {"--hive-table", "-ht"}, description = "hive table")
+ private String hiveTable = "hoodie_sample_test";
+
+ @Parameter(names = {"--hive-user", "-hu"}, description = "hive username")
+ private String hiveUser = "hive";
+
+ @Parameter(names = {"--hive-password", "-hp"}, description = "hive password")
+ private String hivePass = "hive";
+
+ @Parameter(names = {"--hive-url", "-hl"}, description = "hive JDBC URL")
+ private String hiveJdbcUrl = "jdbc:hive2://localhost:10000";
+
+ @Parameter(names = {"--non-partitioned", "-np"}, description = "Use non-partitioned Table")
+ private Boolean nonPartitionedTable = false;
+
+ @Parameter(names = {"--use-multi-partition-keys", "-mp"}, description = "Use Multiple Partition Keys")
+ private Boolean useMultiPartitionKeys = false;
+
+ @Parameter(names = {"--commit-type", "-ct"}, description = "How may commits will run")
+ private String commitType = "overwrite";
+
+ @Parameter(names = {"--help", "-h"}, help = true)
+ public Boolean help = false;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieJavaApp.class);
Review comment:
`HoodieJavaGenerateApp`?
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ Pair<String, String> stdOutErr;
+ if ("overwrite".equals(commitType)) {
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ // Set sleep time by hoodie.hiveserver.time.wait
+ Thread.sleep(getTimeWait());
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ stdOutErr = executeHiveCommand("show tables like '" + hiveTableName + "'");
+ assertTrue(stdOutErr.getLeft().isEmpty(), "Dropped table " + hiveTableName + " exists!");
+ }
+
+ // Run Hoodie Java App
+ String cmd = String.format("%s %s --hive-sync --table-path %s --hive-url %s --table-type %s --hive-table %s" +
+ " --commit-type %s --table-name %s", HOODIE_JAVA_APP, "HoodieJavaGenerateApp", hdfsUrl, HIVE_SERVER_JDBC_URL,
+ tableType, hiveTableName, commitType, hoodieTableName);
+ if (partitionType == PartitionType.MULTI_KEYS_PARTITIONED) {
+ cmd = cmd + " --use-multi-partition-keys";
+ } else if (partitionType == PartitionType.NON_PARTITIONED){
+ cmd = cmd + " --non-partitioned";
+ }
+ executeCommandStringInDocker(ADHOC_1_CONTAINER, cmd, true);
+
+ String snapshotTableName = getSnapshotTableName(tableType, hiveTableName);
+
+ // Ensure table does exist
+ stdOutErr = executeHiveCommand("show tables like '" + snapshotTableName + "'");
+ assertEquals(snapshotTableName, stdOutErr.getLeft(), "Table exists");
+ }
+
+ protected void dropHiveTables(String hiveTableName, String tableType) throws Exception {
+ if (tableType.equals(HoodieTableType.MERGE_ON_READ.name())) {
+ executeHiveCommand("drop table if exists " + hiveTableName + "_rt");
+ executeHiveCommand("drop table if exists " + hiveTableName + "_ro");
+ } else {
+ executeHiveCommand("drop table if exists " + hiveTableName);
+ }
+ }
+
+ protected String getHdfsPath(String hiveTableName) {
+ return "/" + hiveTableName;
+ }
+
+ protected String getSnapshotTableName(String tableType, String hiveTableName) {
+ return tableType.equals(HoodieTableType.MERGE_ON_READ.name())
+ ? hiveTableName + "_rt" : hiveTableName;
+ }
+
+ private int getTimeWait() {
+ int timeWait = 5000;
Review comment:
Can we define a default constant for this?
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ Pair<String, String> stdOutErr;
+ if ("overwrite".equals(commitType)) {
Review comment:
Can we define a constant for multiple usages? IMO, hard code is not a good practice.
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/command/ITTestHoodieSyncCommand.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ.command;
+
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.model.HoodieTableType;
+
+import org.apache.hudi.integ.HoodieTestHiveBase;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Integration test class for HoodieSyncCommand in hudi-cli module.
+ */
+public class ITTestHoodieSyncCommand extends HoodieTestHiveBase {
+
+ private static final String HUDI_CLI_TOOL = HOODIE_WS_ROOT + "/hudi-cli/hudi-cli.sh";
+ private static final String SYNC_VALIDATE_COMMANDS = HOODIE_WS_ROOT + "/docker/demo/sync-validate.commands";
+
+ @Test
+ public void testValidateSync() throws Exception {
+ String hiveTableName = "docker_hoodie_sync_valid_test";
+ String hiveTableName2 = "docker_hoodie_sync_valid_test_2";
+
+ generateDataByHoodieJavaApp(
+ hiveTableName, HoodieTableType.COPY_ON_WRITE.name(), PartitionType.SINGLE_KEY_PARTITIONED, "overwrite", hiveTableName);
+
+ syncHoodieTable(hiveTableName2, "INSERT");
+
+ generateDataByHoodieJavaApp(
+ hiveTableName, HoodieTableType.COPY_ON_WRITE.name(), PartitionType.SINGLE_KEY_PARTITIONED, "append", hiveTableName);
+
+ TestExecStartResultCallback result =
+ executeCommandStringInDocker(ADHOC_1_CONTAINER, HUDI_CLI_TOOL + " --cmdfile " + SYNC_VALIDATE_COMMANDS, true);
+
+ String expected = String.format("Count difference now is (count(%s) - count(%s) == %d. Catch up count is %d",
+ hiveTableName, hiveTableName2, 100, 200);
+ assertTrue(result.getStderr().toString().contains(expected));
+
+ dropHiveTables(hiveTableName, HoodieTableType.COPY_ON_WRITE.name());
+ dropHiveTables(hiveTableName2, HoodieTableType.COPY_ON_WRITE.name());
+ }
+
+ private void syncHoodieTable(String hiveTableName, String op) throws Exception {
+ String cmd = "spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.4 "
Review comment:
Using `StringBuilder ` to build `cmd` is a better choice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#discussion_r454987028
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ Thread.sleep(5000);
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ Pair<String, String> stdOutErr = null;
+ if (commitType == "overwrite") {
Review comment:
Replace `==` to `equal` method
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ Thread.sleep(5000);
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ Pair<String, String> stdOutErr = null;
+ if (commitType == "overwrite") {
+ stdOutErr = executeHiveCommand("show tables like '" + hiveTableName + "'");
+ assertTrue(stdOutErr.getLeft().isEmpty(), "Dropped table " + hiveTableName + " exists!");
+ }
+
+ // Run Hoodie Java App
+ String cmd = HOODIE_JAVA_APP + " HoodieJavaGenerateApp --hive-sync --table-path " + hdfsUrl + " --hive-url " + HIVE_SERVER_JDBC_URL
Review comment:
Let's use `String.format()`?
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ Thread.sleep(5000);
Review comment:
Let's make it configurable?
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/HoodieTestHiveBase.java
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.integ;
+
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.util.collection.Pair;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Base class to run cmd and generate data in hive.
+ */
+public class HoodieTestHiveBase extends ITTestBase {
+
+ protected enum PartitionType {
+ SINGLE_KEY_PARTITIONED, MULTI_KEYS_PARTITIONED, NON_PARTITIONED,
+ }
+
+ /**
+ * A basic integration test that runs HoodieJavaApp to create a sample Hoodie data-set and performs upserts on it.
+ * Hive integration and upsert functionality is checked by running a count query in hive console. TODO: Add
+ * spark-shell test-case
+ */
+ public void generateDataByHoodieJavaApp(String hiveTableName, String tableType, PartitionType partitionType,
+ String commitType, String hoodieTableName) throws Exception {
+
+ String hdfsPath = getHdfsPath(hiveTableName);
+ String hdfsUrl = "hdfs://namenode" + hdfsPath;
+
+ // Drop Table if it exists
+ try {
+ dropHiveTables(hiveTableName, tableType);
+ } catch (AssertionError ex) {
+ // In travis, sometimes, the hivemetastore is not ready even though we wait for the port to be up
+ // Workaround to sleep for 5 secs and retry
+ Thread.sleep(5000);
+ dropHiveTables(hiveTableName, tableType);
+ }
+
+ // Ensure table does not exist
+ Pair<String, String> stdOutErr = null;
Review comment:
No need to initialize it to be `null`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
yanghua commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-659342216
@hddong Please check the reason why Travis failed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hddong commented on pull request #1774: [HUDI-703]Add unit test for HoodieSyncCommand
Posted by GitBox <gi...@apache.org>.
hddong commented on pull request #1774:
URL: https://github.com/apache/hudi/pull/1774#issuecomment-659495533
@yanghua : It's green now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org