You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Kannan Rajah <kr...@maprtech.com> on 2015/02/26 23:43:27 UTC
Is SPARK_CLASSPATH really deprecated?
SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and we
should use spark.executor.extraClassPath instead. But the online
documentation states that spark.executor.extraClassPath is only meant for
backward compatibility.
https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
Which one is right? I have a use case to submit a hbase job from
spark-shell and make it run using YARN. In this case, I need to somehow add
the hbase jars to the classpath of the executor. If I add it to
SPARK_CLASSPATH and export it it works fine. Alternatively, if I set
the spark.executor.extraClassPath in spark-defaults.conf, it works fine.
But the reason I don't like spark-defaults.conf is that I need to hard code
it instead of relying on a script to generate the classpath. I can use a
script in spark-env.sh and set SPARK_CLASSPATH.
Given that compute-classpath uses SPARK_CLASSPATH variable, why is it
marked as deprecated?
--
Kannan
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Todd Nist <ts...@gmail.com>.
Hi Kannan,
I believe you should be able to use the --jars for this when invoke the
spark-shell or perform a spark-submit. Per docs:
--jars JARS Comma-separated list of local jars to include on the
driver
and executor classpaths.
HTH.
-Todd
On Thu, Feb 26, 2015 at 5:43 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and we
> should use spark.executor.extraClassPath instead. But the online
> documentation states that spark.executor.extraClassPath is only meant for
> backward compatibility.
>
> https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
>
> Which one is right? I have a use case to submit a hbase job from
> spark-shell and make it run using YARN. In this case, I need to somehow add
> the hbase jars to the classpath of the executor. If I add it to
> SPARK_CLASSPATH and export it it works fine. Alternatively, if I set
> the spark.executor.extraClassPath in spark-defaults.conf, it works fine.
> But the reason I don't like spark-defaults.conf is that I need to hard code
> it instead of relying on a script to generate the classpath. I can use a
> script in spark-env.sh and set SPARK_CLASSPATH.
>
> Given that compute-classpath uses SPARK_CLASSPATH variable, why is it
> marked as deprecated?
>
> --
> Kannan
>
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Marcelo Vanzin <va...@cloudera.com>.
Just a note for whoever writes the doc, spark.executor.extraClassPath
is *prepended* to the executor's classpath, which is a rather
important distinction. :-)
On Fri, Feb 27, 2015 at 12:21 AM, Patrick Wendell <pw...@gmail.com> wrote:
> I think we need to just update the docs, it is a bit unclear right
> now. At the time, we made it worded fairly sternly because we really
> wanted people to use --jars when we deprecated SPARK_CLASSPATH. But
> there are other types of deployments where there is a legitimate need
> to augment the classpath of every executor.
>
> I think it should probably say something more like
>
> "Extra classpath entries to append to the classpath of executors. This
> is sometimes used in deployment environments where dependencies of
> Spark are present in a specific place on all nodes".
>
> Kannan - if you want to submit I patch I can help review it.
>
> On Thu, Feb 26, 2015 at 8:24 PM, Kannan Rajah <kr...@maprtech.com> wrote:
>> Thanks Marcelo. Do you think it would be useful to make
>> spark.executor.extraClassPath be made to pick up some environment variable
>> that can be set from spark-env.sh? Here is a example.
>>
>> spark-env.sh
>> ------------------
>> executor_extra_cp = get_hbase_jars_for_cp
>> export executor_extra_cp
>>
>> spark-defaults.conf
>> ---------------------
>> spark.executor.extraClassPath = ${executor_extra_cp}
>>
>> This will let us add logic inside get_hbase_jars_for_cp function to pick the
>> right version hbase jars. There could be multiple versions installed on the
>> node.
>>
>>
>>
>> --
>> Kannan
>>
>> On Thu, Feb 26, 2015 at 6:08 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>>
>>> On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kr...@maprtech.com> wrote:
>>> > Also, I would like to know if there is a localization overhead when we
>>> > use
>>> > spark.executor.extraClassPath. Again, in the case of hbase, these jars
>>> > would
>>> > be typically available on all nodes. So there is no need to localize
>>> > them
>>> > from the node where job was submitted. I am wondering if we use the
>>> > SPARK_CLASSPATH approach, then it would not do localization. That would
>>> > be
>>> > an added benefit.
>>> > Please clarify.
>>>
>>> spark.executor.extraClassPath doesn't localize anything. It just
>>> prepends those classpath entries to the usual classpath used to launch
>>> the executor. There's no copying of files or anything, so they're
>>> expected to exist on the nodes.
>>>
>>> It's basically exactly the same as SPARK_CLASSPATH, but broken down to
>>> two options (one for the executors, and one for the driver).
>>>
>>> --
>>> Marcelo
>>
>>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Is SPARK_CLASSPATH really deprecated?
Posted by Taeyun Kim <ta...@innowireless.com>.
spark.executor.extraClassPath is especially useful when the output is
written to HBase, since the data nodes on the cluster have HBase library
jars.
-----Original Message-----
From: Patrick Wendell [mailto:pwendell@gmail.com]
Sent: Friday, February 27, 2015 5:22 PM
To: Kannan Rajah
Cc: Marcelo Vanzin; user@spark.apache.org
Subject: Re: Is SPARK_CLASSPATH really deprecated?
I think we need to just update the docs, it is a bit unclear right now. At
the time, we made it worded fairly sternly because we really wanted people
to use --jars when we deprecated SPARK_CLASSPATH. But there are other types
of deployments where there is a legitimate need to augment the classpath of
every executor.
I think it should probably say something more like
"Extra classpath entries to append to the classpath of executors. This is
sometimes used in deployment environments where dependencies of Spark are
present in a specific place on all nodes".
Kannan - if you want to submit I patch I can help review it.
On Thu, Feb 26, 2015 at 8:24 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> Thanks Marcelo. Do you think it would be useful to make
> spark.executor.extraClassPath be made to pick up some environment
> variable that can be set from spark-env.sh? Here is a example.
>
> spark-env.sh
> ------------------
> executor_extra_cp = get_hbase_jars_for_cp export executor_extra_cp
>
> spark-defaults.conf
> ---------------------
> spark.executor.extraClassPath = ${executor_extra_cp}
>
> This will let us add logic inside get_hbase_jars_for_cp function to
> pick the right version hbase jars. There could be multiple versions
> installed on the node.
>
>
>
> --
> Kannan
>
> On Thu, Feb 26, 2015 at 6:08 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:
>>
>> On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kr...@maprtech.com>
wrote:
>> > Also, I would like to know if there is a localization overhead when
>> > we use spark.executor.extraClassPath. Again, in the case of hbase,
>> > these jars would be typically available on all nodes. So there is
>> > no need to localize them from the node where job was submitted. I
>> > am wondering if we use the SPARK_CLASSPATH approach, then it would
>> > not do localization. That would be an added benefit.
>> > Please clarify.
>>
>> spark.executor.extraClassPath doesn't localize anything. It just
>> prepends those classpath entries to the usual classpath used to
>> launch the executor. There's no copying of files or anything, so
>> they're expected to exist on the nodes.
>>
>> It's basically exactly the same as SPARK_CLASSPATH, but broken down
>> to two options (one for the executors, and one for the driver).
>>
>> --
>> Marcelo
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Patrick Wendell <pw...@gmail.com>.
I think we need to just update the docs, it is a bit unclear right
now. At the time, we made it worded fairly sternly because we really
wanted people to use --jars when we deprecated SPARK_CLASSPATH. But
there are other types of deployments where there is a legitimate need
to augment the classpath of every executor.
I think it should probably say something more like
"Extra classpath entries to append to the classpath of executors. This
is sometimes used in deployment environments where dependencies of
Spark are present in a specific place on all nodes".
Kannan - if you want to submit I patch I can help review it.
On Thu, Feb 26, 2015 at 8:24 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> Thanks Marcelo. Do you think it would be useful to make
> spark.executor.extraClassPath be made to pick up some environment variable
> that can be set from spark-env.sh? Here is a example.
>
> spark-env.sh
> ------------------
> executor_extra_cp = get_hbase_jars_for_cp
> export executor_extra_cp
>
> spark-defaults.conf
> ---------------------
> spark.executor.extraClassPath = ${executor_extra_cp}
>
> This will let us add logic inside get_hbase_jars_for_cp function to pick the
> right version hbase jars. There could be multiple versions installed on the
> node.
>
>
>
> --
> Kannan
>
> On Thu, Feb 26, 2015 at 6:08 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>>
>> On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kr...@maprtech.com> wrote:
>> > Also, I would like to know if there is a localization overhead when we
>> > use
>> > spark.executor.extraClassPath. Again, in the case of hbase, these jars
>> > would
>> > be typically available on all nodes. So there is no need to localize
>> > them
>> > from the node where job was submitted. I am wondering if we use the
>> > SPARK_CLASSPATH approach, then it would not do localization. That would
>> > be
>> > an added benefit.
>> > Please clarify.
>>
>> spark.executor.extraClassPath doesn't localize anything. It just
>> prepends those classpath entries to the usual classpath used to launch
>> the executor. There's no copying of files or anything, so they're
>> expected to exist on the nodes.
>>
>> It's basically exactly the same as SPARK_CLASSPATH, but broken down to
>> two options (one for the executors, and one for the driver).
>>
>> --
>> Marcelo
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Kannan Rajah <kr...@maprtech.com>.
Thanks Marcelo. Do you think it would be useful to make
spark.executor.extraClassPath be made to pick up some environment variable
that can be set from spark-env.sh? Here is a example.
spark-env.sh
------------------
executor_extra_cp = get_hbase_jars_for_cp
export executor_extra_cp
spark-defaults.conf
---------------------
spark.executor.extraClassPath = ${executor_extra_cp}
This will let us add logic inside get_hbase_jars_for_cp function to pick
the right version hbase jars. There could be multiple versions installed on
the node.
--
Kannan
On Thu, Feb 26, 2015 at 6:08 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> > Also, I would like to know if there is a localization overhead when we
> use
> > spark.executor.extraClassPath. Again, in the case of hbase, these jars
> would
> > be typically available on all nodes. So there is no need to localize them
> > from the node where job was submitted. I am wondering if we use the
> > SPARK_CLASSPATH approach, then it would not do localization. That would
> be
> > an added benefit.
> > Please clarify.
>
> spark.executor.extraClassPath doesn't localize anything. It just
> prepends those classpath entries to the usual classpath used to launch
> the executor. There's no copying of files or anything, so they're
> expected to exist on the nodes.
>
> It's basically exactly the same as SPARK_CLASSPATH, but broken down to
> two options (one for the executors, and one for the driver).
>
> --
> Marcelo
>
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> Also, I would like to know if there is a localization overhead when we use
> spark.executor.extraClassPath. Again, in the case of hbase, these jars would
> be typically available on all nodes. So there is no need to localize them
> from the node where job was submitted. I am wondering if we use the
> SPARK_CLASSPATH approach, then it would not do localization. That would be
> an added benefit.
> Please clarify.
spark.executor.extraClassPath doesn't localize anything. It just
prepends those classpath entries to the usual classpath used to launch
the executor. There's no copying of files or anything, so they're
expected to exist on the nodes.
It's basically exactly the same as SPARK_CLASSPATH, but broken down to
two options (one for the executors, and one for the driver).
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Todd Nist <ts...@gmail.com>.
Hi Kannan,
Issues with using --jars make sense. I believe you can set the classpath
via the use the --conf spark.executor.extraClassPath=.... or in your driver
with .set("spark.executor.extraClassPath", ".....")
I believe you are correct with the localize as well as long as your
guaranteed that all nodes have the same versions of said jars.
-Todd
On Thu, Feb 26, 2015 at 8:12 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> There is a usability concern I have with the current way of specifying
> --jars. Imagine a use case like hbase where a lot of jobs need it in its
> classpath. This needs to be set every time. If we use spark.executor.extraClassPath,
> then we just need to set it once But there is no programmatic way to set
> this value, like picking up from an environment variable or by running a
> script that generates classpath. You need to hard code the jars in
> spark-defaults.conf.
>
> Also, I would like to know if there is a localization overhead when we use
> spark.executor.extraClassPath. Again, in the case of hbase, these jars
> would be typically available on all nodes. So there is no need to localize
> them from the node where job was submitted. I am wondering if we use the
> SPARK_CLASSPATH approach, then it would not do localization. That would be
> an added benefit.
> Please clarify.
>
>
>
>
> --
> Kannan
>
> On Thu, Feb 26, 2015 at 4:15 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> SPARK_CLASSPATH is definitely deprecated, but my understanding is that
>> spark.executor.extraClassPath is not, so maybe the documentation needs
>> fixing.
>>
>> I'll let someone who might know otherwise comment, though.
>>
>> On Thu, Feb 26, 2015 at 2:43 PM, Kannan Rajah <kr...@maprtech.com>
>> wrote:
>> > SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and
>> we
>> > should use spark.executor.extraClassPath instead. But the online
>> > documentation states that spark.executor.extraClassPath is only meant
>> for
>> > backward compatibility.
>> >
>> >
>> https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
>> >
>> > Which one is right? I have a use case to submit a hbase job from
>> spark-shell
>> > and make it run using YARN. In this case, I need to somehow add the
>> hbase
>> > jars to the classpath of the executor. If I add it to SPARK_CLASSPATH
>> and
>> > export it it works fine. Alternatively, if I set the
>> > spark.executor.extraClassPath in spark-defaults.conf, it works fine.
>> But the
>> > reason I don't like spark-defaults.conf is that I need to hard code it
>> > instead of relying on a script to generate the classpath. I can use a
>> script
>> > in spark-env.sh and set SPARK_CLASSPATH.
>> >
>> > Given that compute-classpath uses SPARK_CLASSPATH variable, why is it
>> marked
>> > as deprecated?
>> >
>> > --
>> > Kannan
>>
>>
>>
>> --
>> Marcelo
>>
>
>
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Kannan Rajah <kr...@maprtech.com>.
There is a usability concern I have with the current way of specifying
--jars. Imagine a use case like hbase where a lot of jobs need it in its
classpath. This needs to be set every time. If we use
spark.executor.extraClassPath,
then we just need to set it once But there is no programmatic way to set
this value, like picking up from an environment variable or by running a
script that generates classpath. You need to hard code the jars in
spark-defaults.conf.
Also, I would like to know if there is a localization overhead when we use
spark.executor.extraClassPath. Again, in the case of hbase, these jars
would be typically available on all nodes. So there is no need to localize
them from the node where job was submitted. I am wondering if we use the
SPARK_CLASSPATH approach, then it would not do localization. That would be
an added benefit.
Please clarify.
--
Kannan
On Thu, Feb 26, 2015 at 4:15 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> SPARK_CLASSPATH is definitely deprecated, but my understanding is that
> spark.executor.extraClassPath is not, so maybe the documentation needs
> fixing.
>
> I'll let someone who might know otherwise comment, though.
>
> On Thu, Feb 26, 2015 at 2:43 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> > SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and
> we
> > should use spark.executor.extraClassPath instead. But the online
> > documentation states that spark.executor.extraClassPath is only meant for
> > backward compatibility.
> >
> >
> https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
> >
> > Which one is right? I have a use case to submit a hbase job from
> spark-shell
> > and make it run using YARN. In this case, I need to somehow add the hbase
> > jars to the classpath of the executor. If I add it to SPARK_CLASSPATH and
> > export it it works fine. Alternatively, if I set the
> > spark.executor.extraClassPath in spark-defaults.conf, it works fine. But
> the
> > reason I don't like spark-defaults.conf is that I need to hard code it
> > instead of relying on a script to generate the classpath. I can use a
> script
> > in spark-env.sh and set SPARK_CLASSPATH.
> >
> > Given that compute-classpath uses SPARK_CLASSPATH variable, why is it
> marked
> > as deprecated?
> >
> > --
> > Kannan
>
>
>
> --
> Marcelo
>
Re: Is SPARK_CLASSPATH really deprecated?
Posted by Marcelo Vanzin <va...@cloudera.com>.
SPARK_CLASSPATH is definitely deprecated, but my understanding is that
spark.executor.extraClassPath is not, so maybe the documentation needs
fixing.
I'll let someone who might know otherwise comment, though.
On Thu, Feb 26, 2015 at 2:43 PM, Kannan Rajah <kr...@maprtech.com> wrote:
> SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and we
> should use spark.executor.extraClassPath instead. But the online
> documentation states that spark.executor.extraClassPath is only meant for
> backward compatibility.
>
> https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
>
> Which one is right? I have a use case to submit a hbase job from spark-shell
> and make it run using YARN. In this case, I need to somehow add the hbase
> jars to the classpath of the executor. If I add it to SPARK_CLASSPATH and
> export it it works fine. Alternatively, if I set the
> spark.executor.extraClassPath in spark-defaults.conf, it works fine. But the
> reason I don't like spark-defaults.conf is that I need to hard code it
> instead of relying on a script to generate the classpath. I can use a script
> in spark-env.sh and set SPARK_CLASSPATH.
>
> Given that compute-classpath uses SPARK_CLASSPATH variable, why is it marked
> as deprecated?
>
> --
> Kannan
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org