You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zhang, Liyun" <li...@intel.com> on 2017/12/05 05:10:44 UTC
Does any one know where I can upload
spark-2.2.0-bin-hadoop2-without-hive.tgz
Hi all:
Now I am working on HIVE-18150<https://issues.apache.org/jira/browse/HIVE-18150>(Upgrade Spark Version to 2.2.0). I found that I need to upload the spark-2.2.0-bin-hadoop2-without-hive.tgz to a public folder to let hive-qa to download and
test all qfiles relating about spark. Does anyone know where I can upload this spark-2.2.0-bin-hadoop2-without-hive.tgz(nearly 143M). I tried to use Google Driver, but curl xxxx failed.
I need change the url link(http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz<http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-$%7bspark.version%7d-bin-hadoop2-without-hive.tgz>)
$HIVE_SOURCE/itests/pom.xml
<configuration>
<target>
<echo file="target/download.sh">
set -x
/bin/pwd
BASE_DIR=./target
HIVE_ROOT=$BASE_DIR/../../../
DOWNLOAD_DIR=./../thirdparty
download() {
url=$1;
finalName=$2
tarName=$(basename $url)
rm -rf $BASE_DIR/$finalName
if [[ ! -f $DOWNLOAD_DIR/$tarName ]]
then
curl -Sso $DOWNLOAD_DIR/$tarName $url
else
local md5File="$tarName".md5sum
curl -Sso $DOWNLOAD_DIR/$md5File "$url".md5sum
cd $DOWNLOAD_DIR
if type md5sum >/dev/null && ! md5sum -c $md5File; then
curl -Sso $DOWNLOAD_DIR/$tarName $url || return 1
fi
cd -
fi
tar -zxf $DOWNLOAD_DIR/$tarName -C $BASE_DIR
mv $BASE_DIR/spark-${spark.version}-bin-hadoop2-without-hive $BASE_DIR/$finalName
}
mkdir -p $DOWNLOAD_DIR
download "http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz" "spark"
cp -f $HIVE_ROOT/data/conf/spark/log4j2.properties $BASE_DIR/spark/conf/
Best Regards
Kelly Zhang/Zhang,Liyun
RE: Does any one know where I can upload
spark-2.2.0-bin-hadoop2-without-hive.tgz
Posted by "Zhang, Liyun" <li...@intel.com>.
Hi Sahil:
Very thanks, I can download 2.2 pkg in http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-2.2.0-bin-hadoop2-without-hive.tgz!
Best Regards
Kelly Zhang/Zhang,Liyun
-----Original Message-----
From: Sahil Takiar [mailto:takiar.sahil@gmail.com]
Sent: Wednesday, December 6, 2017 9:25 AM
To: dev@hive.apache.org
Subject: Re: Does any one know where I can upload spark-2.2.0-bin-hadoop2-without-hive.tgz
Hey Kelly,
I built and uploaded the new version of the distribution for Spark 2.2.0, it should work now. Let me know if there are any issues.
--Sahil
On Tue, Dec 5, 2017 at 10:44 AM, Sahil Takiar <ta...@gmail.com>
wrote:
> Let me look into how to do this, I'll get back to you soon.
>
> On Mon, Dec 4, 2017 at 9:10 PM, Zhang, Liyun <li...@intel.com>
> wrote:
>
>> Hi all:
>>
>> Now I am working on HIVE-18150<https://issues.apac
>> he.org/jira/browse/HIVE-18150>(Upgrade Spark Version to 2.2.0). I
>> found that I need to upload the
>> spark-2.2.0-bin-hadoop2-without-hive.tgz to a public folder to let
>> hive-qa to download and test all qfiles relating about spark. Does
>> anyone know where I can upload this
>> spark-2.2.0-bin-hadoop2-without-hive.tgz(nearly 143M). I tried to use Google Driver, but curl xxxx failed.
>>
>>
>> I need change the url link(http://d3jw87u4immizc.clo
>> udfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2
>> -without-hive.tgz<http://d3jw87u4immizc.cloudfront.net/
>> spark-tarball/spark-$%7bspark.version%7d-bin-hadoop2-without-hive.tgz
>> >)
>> $HIVE_SOURCE/itests/pom.xml
>> <configuration>
>> <target>
>> <echo file="target/download.sh">
>> set -x
>> /bin/pwd
>> BASE_DIR=./target
>> HIVE_ROOT=$BASE_DIR/../../../
>> DOWNLOAD_DIR=./../thirdparty
>> download() {
>> url=$1;
>> finalName=$2
>> tarName=$(basename $url)
>> rm -rf $BASE_DIR/$finalName
>> if [[ ! -f $DOWNLOAD_DIR/$tarName ]]
>> then
>> curl -Sso $DOWNLOAD_DIR/$tarName $url
>> else
>> local md5File="$tarName".md5sum
>> curl -Sso $DOWNLOAD_DIR/$md5File "$url".md5sum
>> cd $DOWNLOAD_DIR
>> if type md5sum >/dev/null && ! md5sum -c $md5File; then
>> curl -Sso $DOWNLOAD_DIR/$tarName $url || return 1
>> fi
>>
>> cd -
>> fi
>> tar -zxf $DOWNLOAD_DIR/$tarName -C $BASE_DIR
>> mv $BASE_DIR/spark-${spark.version}-bin-hadoop2-without-hive
>> $BASE_DIR/$finalName
>> }
>> mkdir -p $DOWNLOAD_DIR
>> download "http://d3jw87u4immizc.cloudfr
>> ont.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz"
>> "spark"
>> cp -f $HIVE_ROOT/data/conf/spark/log4j2.properties
>> $BASE_DIR/spark/conf/
>>
>>
>>
>>
>> Best Regards
>> Kelly Zhang/Zhang,Liyun
>>
>>
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sahil@gmail.com | (510) 673-0309
>
--
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309
Re: Does any one know where I can upload spark-2.2.0-bin-hadoop2-without-hive.tgz
Posted by Sahil Takiar <ta...@gmail.com>.
Hey Kelly,
I built and uploaded the new version of the distribution for Spark 2.2.0,
it should work now. Let me know if there are any issues.
--Sahil
On Tue, Dec 5, 2017 at 10:44 AM, Sahil Takiar <ta...@gmail.com>
wrote:
> Let me look into how to do this, I'll get back to you soon.
>
> On Mon, Dec 4, 2017 at 9:10 PM, Zhang, Liyun <li...@intel.com>
> wrote:
>
>> Hi all:
>>
>> Now I am working on HIVE-18150<https://issues.apac
>> he.org/jira/browse/HIVE-18150>(Upgrade Spark Version to 2.2.0). I found
>> that I need to upload the spark-2.2.0-bin-hadoop2-without-hive.tgz to a
>> public folder to let hive-qa to download and
>> test all qfiles relating about spark. Does anyone know where I can
>> upload this spark-2.2.0-bin-hadoop2-without-hive.tgz(nearly 143M). I
>> tried to use Google Driver, but curl xxxx failed.
>>
>>
>> I need change the url link(http://d3jw87u4immizc.clo
>> udfront.net/spark-tarball/spark-${spark.version}-bin-hadoop2
>> -without-hive.tgz<http://d3jw87u4immizc.cloudfront.net/
>> spark-tarball/spark-$%7bspark.version%7d-bin-hadoop2-without-hive.tgz>)
>> $HIVE_SOURCE/itests/pom.xml
>> <configuration>
>> <target>
>> <echo file="target/download.sh">
>> set -x
>> /bin/pwd
>> BASE_DIR=./target
>> HIVE_ROOT=$BASE_DIR/../../../
>> DOWNLOAD_DIR=./../thirdparty
>> download() {
>> url=$1;
>> finalName=$2
>> tarName=$(basename $url)
>> rm -rf $BASE_DIR/$finalName
>> if [[ ! -f $DOWNLOAD_DIR/$tarName ]]
>> then
>> curl -Sso $DOWNLOAD_DIR/$tarName $url
>> else
>> local md5File="$tarName".md5sum
>> curl -Sso $DOWNLOAD_DIR/$md5File "$url".md5sum
>> cd $DOWNLOAD_DIR
>> if type md5sum >/dev/null && ! md5sum -c $md5File; then
>> curl -Sso $DOWNLOAD_DIR/$tarName $url || return 1
>> fi
>>
>> cd -
>> fi
>> tar -zxf $DOWNLOAD_DIR/$tarName -C $BASE_DIR
>> mv $BASE_DIR/spark-${spark.version}-bin-hadoop2-without-hive
>> $BASE_DIR/$finalName
>> }
>> mkdir -p $DOWNLOAD_DIR
>> download "http://d3jw87u4immizc.cloudfr
>> ont.net/spark-tarball/spark-${spark.version}-bin-hadoop2-without-hive.tgz"
>> "spark"
>> cp -f $HIVE_ROOT/data/conf/spark/log4j2.properties
>> $BASE_DIR/spark/conf/
>>
>>
>>
>>
>> Best Regards
>> Kelly Zhang/Zhang,Liyun
>>
>>
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sahil@gmail.com | (510) 673-0309
>
--
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309
Re: Does any one know where I can upload spark-2.2.0-bin-hadoop2-without-hive.tgz
Posted by Sahil Takiar <ta...@gmail.com>.
Let me look into how to do this, I'll get back to you soon.
On Mon, Dec 4, 2017 at 9:10 PM, Zhang, Liyun <li...@intel.com> wrote:
> Hi all:
>
> Now I am working on HIVE-18150<https://issues.apache.org/jira/browse/HIVE-
> 18150>(Upgrade Spark Version to 2.2.0). I found that I need to upload the
> spark-2.2.0-bin-hadoop2-without-hive.tgz to a public folder to let
> hive-qa to download and
> test all qfiles relating about spark. Does anyone know where I can
> upload this spark-2.2.0-bin-hadoop2-without-hive.tgz(nearly 143M). I
> tried to use Google Driver, but curl xxxx failed.
>
>
> I need change the url link(http://d3jw87u4immizc.
> cloudfront.net/spark-tarball/spark-${spark.version}-bin-
> hadoop2-without-hive.tgz<http://d3jw87u4immizc.cloudfront.
> net/spark-tarball/spark-$%7bspark.version%7d-bin-hadoop2-without-hive.tgz
> >)
> $HIVE_SOURCE/itests/pom.xml
> <configuration>
> <target>
> <echo file="target/download.sh">
> set -x
> /bin/pwd
> BASE_DIR=./target
> HIVE_ROOT=$BASE_DIR/../../../
> DOWNLOAD_DIR=./../thirdparty
> download() {
> url=$1;
> finalName=$2
> tarName=$(basename $url)
> rm -rf $BASE_DIR/$finalName
> if [[ ! -f $DOWNLOAD_DIR/$tarName ]]
> then
> curl -Sso $DOWNLOAD_DIR/$tarName $url
> else
> local md5File="$tarName".md5sum
> curl -Sso $DOWNLOAD_DIR/$md5File "$url".md5sum
> cd $DOWNLOAD_DIR
> if type md5sum >/dev/null && ! md5sum -c $md5File; then
> curl -Sso $DOWNLOAD_DIR/$tarName $url || return 1
> fi
>
> cd -
> fi
> tar -zxf $DOWNLOAD_DIR/$tarName -C $BASE_DIR
> mv $BASE_DIR/spark-${spark.version}-bin-hadoop2-without-hive
> $BASE_DIR/$finalName
> }
> mkdir -p $DOWNLOAD_DIR
> download "http://d3jw87u4immizc.cloudfront.net/spark-tarball/
> spark-${spark.version}-bin-hadoop2-without-hive.tgz" "spark"
> cp -f $HIVE_ROOT/data/conf/spark/log4j2.properties
> $BASE_DIR/spark/conf/
>
>
>
>
> Best Regards
> Kelly Zhang/Zhang,Liyun
>
>
--
Sahil Takiar
Software Engineer
takiar.sahil@gmail.com | (510) 673-0309