You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/06 07:18:40 UTC
[GitHub] [incubator-hudi] yanghua opened a new pull request #1191:
[HUDI-503] Add hudi test suite documentation into the README file of the
test suite module
yanghua opened a new pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191
## What is the purpose of the pull request
Add hudi test suite documentation into the README file of the test suite module
## Brief change log
- *Add hudi test suite documentation into the README file of the test suite module*
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] sev7e0 commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
sev7e0 commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r363601900
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,291 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
+integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at
+link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite.
+```
+
+## Generating a custom Workload Pattern
+```
+There are 2 ways to generate a workload pattern
+1. Programatically
+Choose to write up the entire DAG of operations programatically, take a look at WorkflowDagGenerator class.
+Once you're ready with the DAG you want to execute, simply pass the class name as follows
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
+...
+2. YAML file
+Choose to write up the entire DAG of operations in YAML, take a look at complex-workload-dag-cow.yaml or
+complex-workload-dag-mor.yaml.
+Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-yaml-path /path/to/your-workflow-dag.yaml
+...
+```
Review comment:
## Generating a custom Workload Pattern
There are 2 ways to generate a workload pattern:
1. Programatically
Choose to write up the entire DAG of operations programatically, take a look at `WorkflowDagGenerator` class. Once you're ready with the DAG you want to execute, simply pass the class name as follows:
```shell
spark-submit
...
...
--class org.apache.hudi.bench.job.HudiTestSuiteJob
--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
...
```
2. YAML file
Choose to write up the entire DAG of operations in YAML, take a look at `complex-workload-dag-cow.yaml` or `complex-workload-dag-mor.yaml`. Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows:
```shell
spark-submit
...
...
--class org.apache.hudi.bench.job.HudiTestSuiteJob
--workload-yaml-path /path/to/your-workflow-dag.yaml
...
```
----
Hey, I think it looks a little clearer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-572843370
@yanghua Left some comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-579385642
@yanghua no worries, happy new year! please take a look at this once you're back from the new year holiday
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-573254806
@yanghua looks good, did you try running it in docker ? Also, can you squash your commits and then I can merge this PR ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r365047159
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,300 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
+integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at
+link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite.
+```
+
+## Generating a custom Workload Pattern
+
+There are 2 ways to generate a workload pattern
+
+ 1.Programatically
+
+Choose to write up the entire DAG of operations programatically, take a look at `WorkflowDagGenerator` class.
+Once you're ready with the DAG you want to execute, simply pass the class name as follows:
+
+```
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
+...
+```
+
+ 2.YAML file
+
+Choose to write up the entire DAG of operations in YAML, take a look at `complex-workload-dag-cow.yaml` or
+`complex-workload-dag-mor.yaml`.
+Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows:
+
+```
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-yaml-path /path/to/your-workflow-dag.yaml
+...
+```
+
+## Building the test suite
+
+The test suite can be found in the `hudi-bench` module. Use the `prepare_integration_suite.sh` script to build
+the test suite, you can provide different parameters to the script.
+
+```
+shell$ ./prepare_integration_suite.sh --help
+Usage: prepare_integration_suite.sh
+ --spark-command, prints the spark command
+ -h, hdfs-version
+ -s, spark version
+ -p, parquet version
+ -a, avro version
+ -s, hive version
+```
+
+```
+shell$ ./prepare_integration_suite.sh
+....
+....
+Final command : mvn clean install -DskipTests
+[INFO] ------------------------------------------------------------------------
+[INFO] Reactor Summary:
+[INFO]
+[INFO] Hudi ............................................... SUCCESS [ 2.749 s]
+[INFO] hudi-common ........................................ SUCCESS [ 12.711 s]
+[INFO] hudi-timeline-service .............................. SUCCESS [ 1.924 s]
+[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 7.203 s]
+[INFO] hudi-client ........................................ SUCCESS [ 10.486 s]
+[INFO] hudi-hive .......................................... SUCCESS [ 5.159 s]
+[INFO] hudi-spark ......................................... SUCCESS [ 34.499 s]
+[INFO] hudi-utilities ..................................... SUCCESS [ 8.626 s]
+[INFO] hudi-cli ........................................... SUCCESS [ 14.921 s]
+[INFO] hudi-bench ......................................... SUCCESS [ 7.706 s]
Review comment:
You might have to re-run and update this with the new package name..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-581785400
> I guess once you rebase this, the build should get fixed (merged your spark upgrade pr)
Yes, we rebase this PR and let Travis recheck it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua edited a comment on issue #1191:
[HUDI-503] Add hudi test suite documentation into the README file of the
test suite module
Posted by GitBox <gi...@apache.org>.
yanghua edited a comment on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-581785400
> I guess once you rebase this, the build should get fixed (merged your spark upgrade pr)
Yes, will rebase this PR and let Travis recheck it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua merged pull request #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-577958035
> @yanghua were you able to fix the build ?
@n3nash Sorry, I still have no time to figure out the root reason. I am in the Chinese New Year holiday now. If you have time, can you help to locate the issue? I believe it is due to bump the Spark version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-571064155
cc @n3nash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r365046918
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,300 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
Review comment:
same
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-574950075
> I verified them, looks ok, please fix the build and I can merge this.
>
> @yanghua
OK, The Travis failure is due to the upgrade of the Spark dependencies. I have rebased the test suite branch. I am trying to figure it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r365046918
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,300 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
Review comment:
same, I would also re-check if the names of the classes are the same..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r365047400
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,300 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
+integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at
+link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite.
+```
+
+## Generating a custom Workload Pattern
+
+There are 2 ways to generate a workload pattern
+
+ 1.Programatically
+
+Choose to write up the entire DAG of operations programatically, take a look at `WorkflowDagGenerator` class.
+Once you're ready with the DAG you want to execute, simply pass the class name as follows:
+
+```
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
+...
+```
+
+ 2.YAML file
+
+Choose to write up the entire DAG of operations in YAML, take a look at `complex-workload-dag-cow.yaml` or
+`complex-workload-dag-mor.yaml`.
+Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows:
+
+```
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-yaml-path /path/to/your-workflow-dag.yaml
+...
+```
+
+## Building the test suite
+
+The test suite can be found in the `hudi-bench` module. Use the `prepare_integration_suite.sh` script to build
+the test suite, you can provide different parameters to the script.
+
+```
+shell$ ./prepare_integration_suite.sh --help
+Usage: prepare_integration_suite.sh
+ --spark-command, prints the spark command
+ -h, hdfs-version
+ -s, spark version
+ -p, parquet version
+ -a, avro version
+ -s, hive version
+```
+
+```
+shell$ ./prepare_integration_suite.sh
+....
+....
+Final command : mvn clean install -DskipTests
+[INFO] ------------------------------------------------------------------------
+[INFO] Reactor Summary:
+[INFO]
+[INFO] Hudi ............................................... SUCCESS [ 2.749 s]
+[INFO] hudi-common ........................................ SUCCESS [ 12.711 s]
+[INFO] hudi-timeline-service .............................. SUCCESS [ 1.924 s]
+[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 7.203 s]
+[INFO] hudi-client ........................................ SUCCESS [ 10.486 s]
+[INFO] hudi-hive .......................................... SUCCESS [ 5.159 s]
+[INFO] hudi-spark ......................................... SUCCESS [ 34.499 s]
+[INFO] hudi-utilities ..................................... SUCCESS [ 8.626 s]
+[INFO] hudi-cli ........................................... SUCCESS [ 14.921 s]
+[INFO] hudi-bench ......................................... SUCCESS [ 7.706 s]
+[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 1.873 s]
+[INFO] hudi-hive-bundle ................................... SUCCESS [ 1.508 s]
+[INFO] hudi-spark-bundle .................................. SUCCESS [ 17.432 s]
+[INFO] hudi-presto-bundle ................................. SUCCESS [ 1.309 s]
+[INFO] hudi-utilities-bundle .............................. SUCCESS [ 18.386 s]
+[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 8.600 s]
+[INFO] hudi-bench-bundle .................................. SUCCESS [ 38.348 s]
+[INFO] hudi-hadoop-docker ................................. SUCCESS [ 2.053 s]
+[INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 0.806 s]
+[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [ 0.302 s]
+[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [ 0.403 s]
+[INFO] hudi-hadoop-history-docker ......................... SUCCESS [ 0.447 s]
+[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 1.534 s]
+[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [ 0.315 s]
+[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 0.407 s]
+[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [ 0.447 s]
+[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 0.410 s]
+[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [ 0.697 s]
+[INFO] hudi-integ-test .................................... SUCCESS [01:02 min]
+[INFO] ------------------------------------------------------------------------
+[INFO] BUILD SUCCESS
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 04:23 min
+[INFO] Finished at: 2019-11-02T23:56:48-07:00
+[INFO] Final Memory: 234M/1582M
+[INFO] ------------------------------------------------------------------------
+```
+
+## Running on the cluster or in your local machine
+Copy over the necessary files and jars that are required to your cluster and then run the following spark-submit
+command after replacing the correct values for the parameters.
+NOTE : The properties-file should have all the necessary information required to ingest into a Hudi dataset. For more
+ information on what properties need to be set, take a look at the test suite section under demo steps.
+```
+shell$ ./prepare_integration_suite.sh --spark-command
+spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 --master prepare_integration_suite.sh --deploy-mode
+--properties-file --class org.apache.hudi.bench.job.HudiTestSuiteJob target/hudi-bench-0.5.1-SNAPSHOT.jar
+--source-class --source-ordering-field --input-base-path --target-base-path --target-table --props --storage-type --payload-class --workload-yaml-path --input-file-size --<deltastreamer-ingest>
+```
+
+## Running through a test-case (local)
+Take a look at the TestHudiTestSuiteJob to check how you can run the entire suite using JUnit.
+
+## Running an end to end test suite in Local Docker environment
+
+```
+docker exec -it adhoc-2 /bin/bash
+# COPY_ON_WRITE tables
+=========================
+## Run the following command to start the test suite
+spark-submit \
Review comment:
please start your docker and try to run both these commands after renaming and package changes to make sure they run fine..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-577509473
@yanghua were you able to fix the build ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r363690393
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,291 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
+class wraps all the functionalities required to run a configurable integration suite.
+```
+
+## Configurations required to run the job
+```
+org.apache.hudi.bench.job.HudiTestSuiteConfig - Config class that drives the behavior of the
+integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at
+link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite.
+```
+
+## Generating a custom Workload Pattern
+```
+There are 2 ways to generate a workload pattern
+1. Programatically
+Choose to write up the entire DAG of operations programatically, take a look at WorkflowDagGenerator class.
+Once you're ready with the DAG you want to execute, simply pass the class name as follows
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-generator-classname org.apache.hudi.bench.dag.scheduler.<your_workflowdaggenerator>
+...
+2. YAML file
+Choose to write up the entire DAG of operations in YAML, take a look at complex-workload-dag-cow.yaml or
+complex-workload-dag-mor.yaml.
+Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows
+spark-submit
+...
+...
+--class org.apache.hudi.bench.job.HudiTestSuiteJob
+--workload-yaml-path /path/to/your-workflow-dag.yaml
+...
+```
Review comment:
Thanks for your suggestion. I have addressed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-574925392
I verified them, looks ok, please fix the build and I can merge this.
@yanghua
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-573271577
> @yanghua looks good, did you try running it in docker ? Also, can you squash your commits and then I can merge this PR ?
Absolutely, I can squash the commits. Sorry, I did not verify those commands in the docker. My local docker env always has some problems. Can you help to verify them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-572890224
@n3nash Have done renaming work. Please have another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-581707602
I guess once you rebase this, the build should get fixed (merged your spark upgrade pr)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] n3nash commented on a change in pull request
#1191: [HUDI-503] Add hudi test suite documentation into the README file of
the test suite module
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#discussion_r365046901
##########
File path: hudi-test-suite/README.md
##########
@@ -0,0 +1,300 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence
+in a release as well as perform large scale performance benchmarks.
+
+# Objectives
+
+1. Test with different versions of core libraries and components such as `hdfs`, `parquet`, `spark`,
+`hive` and `avro`.
+2. Generate different types of workloads across different dimensions such as `payload size`, `number of updates`,
+`number of inserts`, `number of partitions`
+3. Perform multiple types of operations such as `insert`, `bulk_insert`, `upsert`, `compact`, `query`
+4. Support custom post process actions and validations
+
+# High Level Design
+
+The Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
+
+## Workload Generation
+
+This component does the work of generating the workload; `inserts`, `upserts` etc.
+
+## Workload Scheduling
+
+Depending on the type of workload generated, data is either ingested into the target hudi
+dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload
+to be generated/ingested but can require an execution.
+
+## Other actions/operatons
+
+The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
+
+# Usage instructions
+
+
+## Entry class to the test suite
+
+```
+org.apache.hudi.bench.job.HudiTestSuiteJob.java - Entry Point of the hudi test suite job. This
Review comment:
We need to change the package name here, like remove "bench"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] yanghua commented on issue #1191: [HUDI-503] Add
hudi test suite documentation into the README file of the test suite module
Posted by GitBox <gi...@apache.org>.
yanghua commented on issue #1191: [HUDI-503] Add hudi test suite documentation into the README file of the test suite module
URL: https://github.com/apache/incubator-hudi/pull/1191#issuecomment-571541880
cc @n3nash
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services