You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Krishna Sankar <ks...@gmail.com> on 2015/09/23 00:42:34 UTC
HDP 2.3 support for Spark 1.5.x
Guys,
- We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
current wisdom is that it will support the 1.4.x train (which is good, need
DataFrame et al).
- What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on HDP
2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
Cheers & Thanks
<k/>
Re: HDP 2.3 support for Spark 1.5.x
Posted by Krishna Sankar <ks...@gmail.com>.
Thanks Guys. Yep, now I would install 1.5.1 over HDP 2.3, if that works.
Cheers
<k/>
On Mon, Sep 28, 2015 at 9:47 AM, Ted Yu <yu...@gmail.com> wrote:
> Krishna:
> If you want to query ORC files, see the following JIRA:
>
> [SPARK-10623] [SQL] Fixes ORC predicate push-down
>
> which is in the 1.5.1. release.
>
> FYI
>
> On Mon, Sep 28, 2015 at 9:42 AM, Fabien Martin <fa...@gmail.com>
> wrote:
>
>> Hi Krishna,
>>
>> - Take a lokk at *http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>> <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>*
>> - Or you can specify your 1.5.x jar as the Spark one using something
>> like :
>>
>> --conf
>> "spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"
>>
>> The main drawback is :
>>
>> *Known Issues*
>>
>> *Spark YARN ATS integration does not work in this tech preview. You will
>> not see the history of Spark jobs in the Jobs server after a job is
>> finished.*
>>
>> 2015-09-23 1:31 GMT+02:00 Zhan Zhang <zz...@hortonworks.com>:
>>
>>> Hi Krishna,
>>>
>>> For the time being, you can download from upstream, and it should be
>>> running OK for HDP2.3. For hdp specific problem, you can ask in
>>> Hortonworks forum.
>>>
>>> Thanks.
>>>
>>> Zhan Zhang
>>>
>>> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ks...@gmail.com> wrote:
>>>
>>> Guys,
>>>
>>> - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
>>> current wisdom is that it will support the 1.4.x train (which is good, need
>>> DataFrame et al).
>>> - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
>>> HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>>>
>>> Cheers & Thanks
>>> <k/>
>>>
>>>
>>>
>>
>
Re: HDP 2.3 support for Spark 1.5.x
Posted by Ted Yu <yu...@gmail.com>.
Krishna:
If you want to query ORC files, see the following JIRA:
[SPARK-10623] [SQL] Fixes ORC predicate push-down
which is in the 1.5.1. release.
FYI
On Mon, Sep 28, 2015 at 9:42 AM, Fabien Martin <fa...@gmail.com>
wrote:
> Hi Krishna,
>
> - Take a lokk at *http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
> <http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>*
> - Or you can specify your 1.5.x jar as the Spark one using something
> like :
>
> --conf
> "spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"
>
> The main drawback is :
>
> *Known Issues*
>
> *Spark YARN ATS integration does not work in this tech preview. You will
> not see the history of Spark jobs in the Jobs server after a job is
> finished.*
>
> 2015-09-23 1:31 GMT+02:00 Zhan Zhang <zz...@hortonworks.com>:
>
>> Hi Krishna,
>>
>> For the time being, you can download from upstream, and it should be
>> running OK for HDP2.3. For hdp specific problem, you can ask in
>> Hortonworks forum.
>>
>> Thanks.
>>
>> Zhan Zhang
>>
>> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ks...@gmail.com> wrote:
>>
>> Guys,
>>
>> - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
>> current wisdom is that it will support the 1.4.x train (which is good, need
>> DataFrame et al).
>> - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
>> HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>>
>> Cheers & Thanks
>> <k/>
>>
>>
>>
>
Re: HDP 2.3 support for Spark 1.5.x
Posted by Fabien Martin <fa...@gmail.com>.
Hi Krishna,
- Take a lokk at
*http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
<http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/>*
- Or you can specify your 1.5.x jar as the Spark one using something
like :
--conf
"spark.yarn.jar=hdfs://master:8020/spark-assembly-1.5.0-hadoop2.6.0.jar"
The main drawback is :
*Known Issues*
*Spark YARN ATS integration does not work in this tech preview. You will
not see the history of Spark jobs in the Jobs server after a job is
finished.*
2015-09-23 1:31 GMT+02:00 Zhan Zhang <zz...@hortonworks.com>:
> Hi Krishna,
>
> For the time being, you can download from upstream, and it should be
> running OK for HDP2.3. For hdp specific problem, you can ask in
> Hortonworks forum.
>
> Thanks.
>
> Zhan Zhang
>
> On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ks...@gmail.com> wrote:
>
> Guys,
>
> - We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The
> current wisdom is that it will support the 1.4.x train (which is good, need
> DataFrame et al).
> - What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on
> HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
>
> Cheers & Thanks
> <k/>
>
>
>
Re: HDP 2.3 support for Spark 1.5.x
Posted by Zhan Zhang <zz...@hortonworks.com>.
Hi Krishna,
For the time being, you can download from upstream, and it should be running OK for HDP2.3. For hdp specific problem, you can ask in Hortonworks forum.
Thanks.
Zhan Zhang
On Sep 22, 2015, at 3:42 PM, Krishna Sankar <ks...@gmail.com>> wrote:
Guys,
* We have HDP 2.3 installed just now. It comes with Spark 1.3.x. The current wisdom is that it will support the 1.4.x train (which is good, need DataFrame et al).
* What is the plan to support Spark 1.5.x ? Can we install 1.5.0 on HDP 2.3 ? Or will Spark 1.5.x support be in HDP 2.3.x and if so ~when ?
Cheers & Thanks
<k/>