[jira] [Commented] (SPARK-19836) Customizable remote repository url for hive versions unit test

["Elek, Marton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897172#comment-15897172 ] 

Elek, Marton commented on SPARK-19836:
--------------------------------------

I am working on a patch for this one. Could you please assign it to me, if the proposal is acceptable?

> Customizable remote repository url for hive versions unit test
> --------------------------------------------------------------
>
>                 Key: SPARK-19836
>                 URL: https://issues.apache.org/jira/browse/SPARK-19836
>             Project: Spark
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 2.1.0
>            Reporter: Elek, Marton
>              Labels: ivy, unittest
>
> When the VersionSuite test runs from sql/hive it downloads different versions from hive.
> Unfortunately the IsolatedClientClassloader (which is used by the VersionSuite) uses hardcoded fix repositories:
> {code}
>     val classpath = quietly {
>       SparkSubmitUtils.resolveMavenCoordinates(
>         hiveArtifacts.mkString(","),
>         SparkSubmitUtils.buildIvySettings(
>           Some("http://www.datanucleus.org/downloads/maven2"),
>           ivyPath),
>         exclusions = version.exclusions)
>     }
> {code}
> The problem is with the hard-coded repositories:
>  1. it's hard to run unit tests in an environment where only one internal maven repository is available (and central/datanucleus is not)
>  2. it's impossible to run unit tests against custom built hive/hadoop artifacts (which are not available from the central repository)
> VersionSuite has already a specific SPARK_VERSIONS_SUITE_IVY_PATH environment variable to define a custom local repository as ivy cache.
> I suggest to add an additional environment variable (SPARK_VERSIONS_SUITE_IVY_REPOSITORIES to the HiveClientBuilder.scala), to make it possible adding new remote repositories for testing the different hive versions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org