You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by pr...@gmail.com on 2017/02/23 01:02:46 UTC
RDD blocks on Spark Driver
Hello,
Had a question. When I look at the executors tab in Spark UI, I notice that some RDD blocks are assigned to the driver as well. Can someone please tell me why?
Thanks for the help.
Re: RDD blocks on Spark Driver
Posted by Daniel Santana <da...@everymundo.com>.
Hi you all
I'm seeing the same thing over here. EMR as well.
1 Master + 2 Slaves (But it is using only one)
Spark Submit command:
$SPARK_HOME/bin/spark-submit \
--class com.mydomain.myappname.streaming.MyClass \
--master yarn \
--deploy-mode cluster \
--conf spark.yarn.max.executor.failures=2 \
--conf spark.yarn.am.attemptFailuresValidityInterval='1h' \
--conf spark.yarn.submit.waitAppCompletion=false \
--conf spark.streaming.receiver.writeAheadLog.enable=true \
--num-executors 6 \
--executor-cores 2 \
--supervise \
./my-uber-jar-file.jar
Does anybody have a clue?
Look:
Best,
*Daniel Santana*
DataCORE Lead Engineer
EVERY*MUNDO*
25 SE 2nd Ave., Suite 900
Miami, FL 33131 USA
+1 (305) 375-0045
EveryMundo.com <http://www.everymundo.com/#whoweare>
*Confidentiality Notice: *This email and any files transmitted with it are
confidential and intended solely for the use of the individual or entity to
whom they are addressed. If you have received this email in error, please
notify the system manager.
On Wed, Mar 1, 2017 at 12:35 AM, Prithish <pr...@gmail.com> wrote:
> This is the command I am running:
>
> spark-submit --deploy-mode cluster --master yarn --class com.myorg.myApp
> s3://my-bucket/myapp-0.1.jar
>
> On Wed, Mar 1, 2017 at 12:22 AM, Jonathan Kelly <jo...@gmail.com>
> wrote:
>
>> Prithish,
>>
>> It would be helpful for you to share the spark-submit command you are
>> running.
>>
>> ~ Jonathan
>>
>> On Sun, Feb 26, 2017 at 8:29 AM Prithish <pr...@gmail.com> wrote:
>>
>>> Thanks for the responses, I am running this on Amazon EMR which runs the
>>> Yarn cluster manager.
>>>
>>> On Sat, Feb 25, 2017 at 4:45 PM, liangyhgood@gmail.com <
>>> liangyhgood@gmail.com> wrote:
>>>
>>> Hi,
>>> I think you are using the local model of Spark. There are
>>> mainly four models, which are local, standalone, yarn and
>>> Mesos. Also, "blocks" is relative to hdfs, "partitions" is
>>> relative to spark.
>>>
>>> liangyihuai
>>>
>>> ---Original---
>>> *From:* "Jacek Laskowski "<ja...@japila.pl>
>>> *Date:* 2017/2/25 02:45:20
>>> *To:* "prithish"<pr...@gmail.com>;
>>> *Cc:* "user"<us...@spark.apache.org>;
>>> *Subject:* Re: RDD blocks on Spark Driver
>>>
>>> Hi,
>>>
>>> Guess you're use local mode which has only one executor called driver.
>>> Is my guessing correct?
>>>
>>> Jacek
>>>
>>> On 23 Feb 2017 2:03 a.m., <pr...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Had a question. When I look at the executors tab in Spark UI, I notice
>>> that some RDD blocks are assigned to the driver as well. Can someone please
>>> tell me why?
>>>
>>> Thanks for the help.
>>>
>>>
>>>
>
Re: RDD blocks on Spark Driver
Posted by Prithish <pr...@gmail.com>.
This is the command I am running:
spark-submit --deploy-mode cluster --master yarn --class com.myorg.myApp
s3://my-bucket/myapp-0.1.jar
On Wed, Mar 1, 2017 at 12:22 AM, Jonathan Kelly <jo...@gmail.com>
wrote:
> Prithish,
>
> It would be helpful for you to share the spark-submit command you are
> running.
>
> ~ Jonathan
>
> On Sun, Feb 26, 2017 at 8:29 AM Prithish <pr...@gmail.com> wrote:
>
>> Thanks for the responses, I am running this on Amazon EMR which runs the
>> Yarn cluster manager.
>>
>> On Sat, Feb 25, 2017 at 4:45 PM, liangyhgood@gmail.com <
>> liangyhgood@gmail.com> wrote:
>>
>> Hi,
>> I think you are using the local model of Spark. There
>> are mainly four models, which are local, standalone, yarn
>> and Mesos. Also, "blocks" is relative to hdfs, "partitions"
>> is relative to spark.
>>
>> liangyihuai
>>
>> ---Original---
>> *From:* "Jacek Laskowski "<ja...@japila.pl>
>> *Date:* 2017/2/25 02:45:20
>> *To:* "prithish"<pr...@gmail.com>;
>> *Cc:* "user"<us...@spark.apache.org>;
>> *Subject:* Re: RDD blocks on Spark Driver
>>
>> Hi,
>>
>> Guess you're use local mode which has only one executor called driver. Is
>> my guessing correct?
>>
>> Jacek
>>
>> On 23 Feb 2017 2:03 a.m., <pr...@gmail.com> wrote:
>>
>> Hello,
>>
>> Had a question. When I look at the executors tab in Spark UI, I notice
>> that some RDD blocks are assigned to the driver as well. Can someone please
>> tell me why?
>>
>> Thanks for the help.
>>
>>
>>
Re: RDD blocks on Spark Driver
Posted by Jonathan Kelly <jo...@gmail.com>.
Prithish,
It would be helpful for you to share the spark-submit command you are
running.
~ Jonathan
On Sun, Feb 26, 2017 at 8:29 AM Prithish <pr...@gmail.com> wrote:
> Thanks for the responses, I am running this on Amazon EMR which runs the
> Yarn cluster manager.
>
> On Sat, Feb 25, 2017 at 4:45 PM, liangyhgood@gmail.com <
> liangyhgood@gmail.com> wrote:
>
> Hi,
>
> I think you are using the local model of Spark. There are mainly four models, which are local, standalone, yarn and Mesos. Also, "blocks" is relative to hdfs, "partitions" is relative to spark.
>
> liangyihuai
>
> ---Original---
> *From:* "Jacek Laskowski "<ja...@japila.pl>
> *Date:* 2017/2/25 02:45:20
> *To:* "prithish"<pr...@gmail.com>;
> *Cc:* "user"<us...@spark.apache.org>;
> *Subject:* Re: RDD blocks on Spark Driver
>
> Hi,
>
> Guess you're use local mode which has only one executor called driver. Is
> my guessing correct?
>
> Jacek
>
> On 23 Feb 2017 2:03 a.m., <pr...@gmail.com> wrote:
>
> Hello,
>
> Had a question. When I look at the executors tab in Spark UI, I notice
> that some RDD blocks are assigned to the driver as well. Can someone please
> tell me why?
>
> Thanks for the help.
>
>
>
Re: RDD blocks on Spark Driver
Posted by Prithish <pr...@gmail.com>.
Thanks for the responses, I am running this on Amazon EMR which runs the
Yarn cluster manager.
On Sat, Feb 25, 2017 at 4:45 PM, liangyhgood@gmail.com <
liangyhgood@gmail.com> wrote:
> Hi,
> I think you are using the local model of Spark. There
> are mainly four models, which are local, standalone, yarn
> and Mesos. Also, "blocks" is relative to hdfs, "partitions"
> is relative to spark.
>
> liangyihuai
>
> ---Original---
> *From:* "Jacek Laskowski "<ja...@japila.pl>
> *Date:* 2017/2/25 02:45:20
> *To:* "prithish"<pr...@gmail.com>;
> *Cc:* "user"<us...@spark.apache.org>;
> *Subject:* Re: RDD blocks on Spark Driver
>
> Hi,
>
> Guess you're use local mode which has only one executor called driver. Is
> my guessing correct?
>
> Jacek
>
> On 23 Feb 2017 2:03 a.m., <pr...@gmail.com> wrote:
>
>> Hello,
>>
>> Had a question. When I look at the executors tab in Spark UI, I notice
>> that some RDD blocks are assigned to the driver as well. Can someone please
>> tell me why?
>>
>> Thanks for the help.
>>
>
Re: RDD blocks on Spark Driver
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Guess you're use local mode which has only one executor called driver. Is
my guessing correct?
Jacek
On 23 Feb 2017 2:03 a.m., <pr...@gmail.com> wrote:
> Hello,
>
> Had a question. When I look at the executors tab in Spark UI, I notice
> that some RDD blocks are assigned to the driver as well. Can someone please
> tell me why?
>
> Thanks for the help.
>