[jira] [Updated] (HIVE-29483) Implement an integration testing mechanism with real Tez on Yarn including resource localization

["László Bodor (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-29483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

László Bodor updated HIVE-29483:
--------------------------------
    Fix Version/s: 4.3.0

> Implement an integration testing mechanism with real Tez on Yarn including resource localization
> ------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-29483
>                 URL: https://issues.apache.org/jira/browse/HIVE-29483
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: László Bodor
>            Assignee: Hemanth Umashankar
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> Follow-up for HIVE-29470
> {code:java}
> implement an integration testing mechanism (inspired by current qtest drivers maybe), that can utilize a real mini yarn cluster, that relies on the localized hive-exec jar (and figure out first how the current TestMiniTezCliDriver works and all tests pass on that driver with the above hack patch in place) <-- nice-to-have long-term
> {code}
> history recap, given this POC patch: https://github.com/abstractdog/hive/commit/06ce23fb04bb2e862d0cf511f5ac214ce1e6bee9
> on a classic yarn cluster, this patch makes all hive queries fail, given that the hive-exec.jar (referred to as app jar) localization is skipped, so when a tez job starts (in a yarn application), that jar is not present, so it immediately fails with some kind of ClassNotFoundException kinda problem
> the implemented testing here is definitely a heavy, integrated testing one, so I can imagine that it won't included into the precommit phase, but users must be able to start a pseudo-distributed hadoop/yarn/hdfs cluster and run tez jobs against that, because apparently, the mini clusters won't cover some cases



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)