[jira] [Updated] (HIVE-14735) Build Infra: Spark artifacts download takes a long time

["Zoltan Haindrich (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zoltan Haindrich updated HIVE-14735:
------------------------------------
    Attachment: HIVE-14735.1.patch

[~spena] it's good to know these things...i've tryed this out - because in gradle this would be easy...i assumed maven can do it too...well, it seems it does! but it needs quite a bunch of xml to do something like this ;)

i've experimented with it...and it looks like it works - i would like to submit a test ptest run to check that everything is all right - but since my own server serves this spark related maven repo, I don't really want it to go in ;)

i've published a preliminary "conceptional" repackaging tool here:
https://github.com/kgyrtkirk/hive-14735

[~spena], you might want to try it out locally, i've sketched a readme in that git repo - hope it helps evaluating this maven option.

I think this will eventually work...download/unpack/etc is done by maven plugins which should be highly portable.

notes:

* {{mvn clean}} clears the unpacked things - which is good
* unpacking a new version doesn't remove the old files, just pastes the new tree on top of it...but in case someone changes distinct branches I think he will use {{mvn clean}} or a harder {{git clean -dfx}} - so it should be ok
* there is a log4j2 properties filewhich gets copied into this unpacked directory...it can be included in the artifact...or keep it like this?


> Build Infra: Spark artifacts download takes a long time
> -------------------------------------------------------
>
>                 Key: HIVE-14735
>                 URL: https://issues.apache.org/jira/browse/HIVE-14735
>             Project: Hive
>          Issue Type: Bug
>          Components: Build Infrastructure
>            Reporter: Vaibhav Gumashta
>            Assignee: Zoltan Haindrich
>         Attachments: HIVE-14735.1.patch
>
>
> In particular this command:
> {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)