You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Fengyun RAO <ra...@gmail.com> on 2014/10/15 15:29:21 UTC
Re: How to add HBase dependencies and conf with spark-submit?
+user@hbase
2014-10-15 20:48 GMT+08:00 Fengyun RAO <ra...@gmail.com>:
> We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an
> HBase table in Spark program.
>
> I notice there are:
> spark.driver.extraClassPath
> spark.executor.extraClassPathproperties to manage extra ClassPath, over
> even an deprecated SPARK_CLASSPATH.
>
> The problem is what classpath or jars should we append?
> I can simplely add the whole `hbase classpath`, which is huge,
> but this leads to dependencies conflict, e.g. HBase uses guava-12 while
> Spark uses guava-14.
>
>
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Soumitra Kumar <ku...@gmail.com>.
Great, it worked.
I don't have an answer what is special about SPARK_CLASSPATH vs --jars, just found the working setting through trial an error.
----- Original Message -----
From: "Fengyun RAO" <ra...@gmail.com>
To: "Soumitra Kumar" <ku...@gmail.com>
Cc: user@spark.apache.org, user@hbase.apache.org
Sent: Thursday, October 16, 2014 12:50:01 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
Thanks, Soumitra Kumar,
I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add hbase-protocol.jar , hbase-common.jar , hbase-client.jar , htrace-core.jar in --jar, but it did work.
Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf directory.
2014-10-15 22:39 GMT+08:00 Soumitra Kumar < kumar.soumitra@gmail.com > :
I am writing to HBase, following are my options:
export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
spark-submit \
--jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \
----- Original Message -----
From: "Fengyun RAO" < raofengyun@gmail.com >
To: user@spark.apache.org , user@hbase.apache.org
Sent: Wednesday, October 15, 2014 6:29:21 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
+user@hbase
2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program.
I notice there are:
spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
The problem is what classpath or jars should we append?
I can simplely add the whole `hbase classpath`, which is huge,
but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Soumitra Kumar <ku...@gmail.com>.
Great, it worked.
I don't have an answer what is special about SPARK_CLASSPATH vs --jars, just found the working setting through trial an error.
----- Original Message -----
From: "Fengyun RAO" <ra...@gmail.com>
To: "Soumitra Kumar" <ku...@gmail.com>
Cc: user@spark.apache.org, user@hbase.apache.org
Sent: Thursday, October 16, 2014 12:50:01 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
Thanks, Soumitra Kumar,
I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add hbase-protocol.jar , hbase-common.jar , hbase-client.jar , htrace-core.jar in --jar, but it did work.
Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf directory.
2014-10-15 22:39 GMT+08:00 Soumitra Kumar < kumar.soumitra@gmail.com > :
I am writing to HBase, following are my options:
export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
spark-submit \
--jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \
----- Original Message -----
From: "Fengyun RAO" < raofengyun@gmail.com >
To: user@spark.apache.org , user@hbase.apache.org
Sent: Wednesday, October 15, 2014 6:29:21 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
+user@hbase
2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program.
I notice there are:
spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
The problem is what classpath or jars should we append?
I can simplely add the whole `hbase classpath`, which is huge,
but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Fengyun RAO <ra...@gmail.com>.
Thanks, Soumitra Kumar,
I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add
hbase-protocol.jar, hbase-common.jar, hbase-client.jar, htrace-core.jar in
--jar, but it did work.
Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf
directory.
2014-10-15 22:39 GMT+08:00 Soumitra Kumar <ku...@gmail.com>:
> I am writing to HBase, following are my options:
>
> export
> SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
>
> spark-submit \
> --jars
> /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar
> \
>
> ----- Original Message -----
> From: "Fengyun RAO" <ra...@gmail.com>
> To: user@spark.apache.org, user@hbase.apache.org
> Sent: Wednesday, October 15, 2014 6:29:21 AM
> Subject: Re: How to add HBase dependencies and conf with spark-submit?
>
>
> +user@hbase
>
>
>
> 2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
>
>
>
> We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an
> HBase table in Spark program.
>
>
>
> I notice there are:
>
> spark.driver.extraClassPath spark.executor.extraClassPath properties to
> manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
>
>
> The problem is what classpath or jars should we append?
> I can simplely add the whole `hbase classpath`, which is huge,
> but this leads to dependencies conflict, e.g. HBase uses guava-12 while
> Spark uses guava-14.
>
>
>
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Fengyun RAO <ra...@gmail.com>.
Thanks, Soumitra Kumar,
I didn’t know why you put hbase-protocol.jar in SPARK_CLASSPATH, while add
hbase-protocol.jar, hbase-common.jar, hbase-client.jar, htrace-core.jar in
--jar, but it did work.
Actually, I put all these four jars in SPARK_CLASSPATH along with HBase conf
directory.
2014-10-15 22:39 GMT+08:00 Soumitra Kumar <ku...@gmail.com>:
> I am writing to HBase, following are my options:
>
> export
> SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
>
> spark-submit \
> --jars
> /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar
> \
>
> ----- Original Message -----
> From: "Fengyun RAO" <ra...@gmail.com>
> To: user@spark.apache.org, user@hbase.apache.org
> Sent: Wednesday, October 15, 2014 6:29:21 AM
> Subject: Re: How to add HBase dependencies and conf with spark-submit?
>
>
> +user@hbase
>
>
>
> 2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
>
>
>
> We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an
> HBase table in Spark program.
>
>
>
> I notice there are:
>
> spark.driver.extraClassPath spark.executor.extraClassPath properties to
> manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
>
>
> The problem is what classpath or jars should we append?
> I can simplely add the whole `hbase classpath`, which is huge,
> but this leads to dependencies conflict, e.g. HBase uses guava-12 while
> Spark uses guava-14.
>
>
>
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Soumitra Kumar <ku...@gmail.com>.
I am writing to HBase, following are my options:
export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
spark-submit \
--jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \
----- Original Message -----
From: "Fengyun RAO" <ra...@gmail.com>
To: user@spark.apache.org, user@hbase.apache.org
Sent: Wednesday, October 15, 2014 6:29:21 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
+user@hbase
2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program.
I notice there are:
spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
The problem is what classpath or jars should we append?
I can simplely add the whole `hbase classpath`, which is huge,
but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to add HBase dependencies and conf with spark-submit?
Posted by Soumitra Kumar <ku...@gmail.com>.
I am writing to HBase, following are my options:
export SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar
spark-submit \
--jars /opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar,/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core.jar \
----- Original Message -----
From: "Fengyun RAO" <ra...@gmail.com>
To: user@spark.apache.org, user@hbase.apache.org
Sent: Wednesday, October 15, 2014 6:29:21 AM
Subject: Re: How to add HBase dependencies and conf with spark-submit?
+user@hbase
2014-10-15 20:48 GMT+08:00 Fengyun RAO < raofengyun@gmail.com > :
We use Spark 1.1, and HBase 0.98.1-cdh5.1.0, and need to read and write an HBase table in Spark program.
I notice there are:
spark.driver.extraClassPath spark.executor.extraClassPath properties to manage extra ClassPath, over even an deprecated SPARK_CLASSPATH.
The problem is what classpath or jars should we append?
I can simplely add the whole `hbase classpath`, which is huge,
but this leads to dependencies conflict, e.g. HBase uses guava-12 while Spark uses guava-14.