[GitHub] [incubator-hudi] n3nash edited a comment on issue #1100: [HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

[GitBox <gi...@apache.org>]

n3nash edited a comment on issue #1100:
URL: https://github.com/apache/incubator-hudi/pull/1100#issuecomment-618113749


   @yanghua @bvaradar @vinothchandar I've fixed this PR since it was failing builds due to multiple pom issues, I've rebased the code from the last time we did this (lots of code has changed naturally) and cleaned up some of the code. 
   
   This test suite now has test cases that one can use to run end to end tests in junit. At the moment, the test suite does not run in docker due to Spark 2.4 bringing in Hive 1.x dependencies and our code using Hive 2.x to spin up local hive server nodes. There is a hacky approach that we are using at Uber which will be upstreamed in the next couple of weeks (https://issues.apache.org/jira/browse/HUDI-830) and as part of that we can discuss how to solve it (the right way to solve without any hacks is to move to spark 3.x since they upgraded the Hive libs there but that might take a while etc). Once that is done, we can even run this as an integration test.
   
   The following is my suggestion for a plan for this : 
   1) Land this PR which provides an initial test suite to test basic end to end functionality. This allows folks to atleast start using this framework to test large PRs but spending minimal amount of time to enhance the test suite (especially since lots of refactoring is happening). Last I checked, @yanghua I think you went over the PR and were okay to merge. @vinothchandar @bvaradar unless we have major concerns on the PR, we can merge it and then take incremental pr's
   2) There are many tickets under HUDI-289, all the enhancements to the test suite from Uber will follow in subsequent PR's in the next 2-3 weeks. I'm working with the necessary folks along with some of the enhancements that I want to do in the coming weeks.
   3) @yanghua need you to lead the Azure pipelines for the test suite and other tickets assigned to you under the umbrella ticket.
   
   Additionally, since the test suite actually tests all end to end functionality, the main class tests take a while to run. Also, without the test suite we are nearing the max time allowed to run unit tests on travis (50 mins), so I've separated out the test suite unit tests into a different job, just like the integration tests.  
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org