You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by se...@nomura.com on 2016/01/07 17:51:25 UTC
Question in rdd caching in memory using persist
Hi,
After I called rdd.persist(MEMORY_ONLY_SER), I see the driver listed as one of the ‘executors’ participating in holding the partitions of the rdd in memory, the memory usage shown against the driver is 0. This I see in the storage tab of the spark ui.
Why is the driver shown on the ui ? Will it ever hold rdd partitions when caching.
-regards
Seemanto Barua
PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
RE: Question in rdd caching in memory using persist
Posted by se...@nomura.com.
Attached is the screen shot of the storage tab details for the cached rdd. The host highlighted and at the end of the list is driver machine.
[cid:image001.png@01D1496C.FD79AB40]
-regards
Seemanto Barua
From: Barua, Seemanto (US)
Sent: Thursday, January 07, 2016 12:43 PM
To: 'premsure542@gmail.com'
Cc: 'user@spark.apache.org'
Subject: Re: Question in rdd caching in memory using persist
I have a standalone cluster. spark version is 1.3.1
From: Prem Sure [mailto:premsure542@gmail.com]
Sent: Thursday, January 07, 2016 12:32 PM
To: Barua, Seemanto (US)
Cc: spark users <us...@spark.apache.org>>
Subject: Re: Question in rdd caching in memory using persist
are you running standalone - local mode or cluster mode. executor and driver existance differ based on setup type. snapshot of your env UI would be helpful to say
On Thu, Jan 7, 2016 at 11:51 AM, <se...@nomura.com>> wrote:
Hi,
After I called rdd.persist(MEMORY_ONLY_SER), I see the driver listed as one of the ‘executors’ participating in holding the partitions of the rdd in memory, the memory usage shown against the driver is 0. This I see in the storage tab of the spark ui.
Why is the driver shown on the ui ? Will it ever hold rdd partitions when caching.
-regards
Seemanto Barua
PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
Re: Question in rdd caching in memory using persist
Posted by se...@nomura.com.
I have a standalone cluster. spark version is 1.3.1
From: Prem Sure [mailto:premsure542@gmail.com]
Sent: Thursday, January 07, 2016 12:32 PM
To: Barua, Seemanto (US)
Cc: spark users <us...@spark.apache.org>
Subject: Re: Question in rdd caching in memory using persist
are you running standalone - local mode or cluster mode. executor and driver existance differ based on setup type. snapshot of your env UI would be helpful to say
On Thu, Jan 7, 2016 at 11:51 AM, <se...@nomura.com>> wrote:
Hi,
After I called rdd.persist(MEMORY_ONLY_SER), I see the driver listed as one of the ‘executors’ participating in holding the partitions of the rdd in memory, the memory usage shown against the driver is 0. This I see in the storage tab of the spark ui.
Why is the driver shown on the ui ? Will it ever hold rdd partitions when caching.
-regards
Seemanto Barua
PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
PLEASE READ: This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please delete it and all copies from your system, destroy any hard copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Nomura Holding America Inc., Nomura Securities International, Inc, and their respective subsidiaries each reserve the right to monitor all e-mail communications through its networks. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state the views of such entity. Unless otherwise stated, any pricing information in this message is indicative only, is subject to change and does not constitute an offer to deal at any price quoted. Any reference to the terms of executed transactions should be treated as preliminary only and subject to our formal written confirmation.
Re: Question in rdd caching in memory using persist
Posted by Prem Sure <pr...@gmail.com>.
are you running standalone - local mode or cluster mode. executor and
driver existance differ based on setup type. snapshot of your env UI would
be helpful to say
On Thu, Jan 7, 2016 at 11:51 AM, <se...@nomura.com> wrote:
> Hi,
>
>
>
> After I called rdd.persist(*MEMORY_ONLY_SER*), I see the driver listed as one of the ‘executors’ participating in holding the partitions of the rdd in memory, the memory usage shown against the driver is 0. This I see in the storage tab of the spark ui.
>
> Why is the driver shown on the ui ? Will it ever hold rdd partitions when
> caching.
>
>
>
>
>
> *-regards*
>
> *Seemanto Barua*
>
>
>
>
>
> PLEASE READ: This message is for the named person's use only. It may
> contain confidential, proprietary or legally privileged information. No
> confidentiality or privilege is waived or lost by any mistransmission. If
> you receive this message in error, please delete it and all copies from
> your system, destroy any hard copies and notify the sender. You must not,
> directly or indirectly, use, disclose, distribute, print, or copy any part
> of this message if you are not the intended recipient. Nomura Holding
> America Inc., Nomura Securities International, Inc, and their respective
> subsidiaries each reserve the right to monitor all e-mail communications
> through its networks. Any views expressed in this message are those of the
> individual sender, except where the message states otherwise and the sender
> is authorized to state the views of such entity. Unless otherwise stated,
> any pricing information in this message is indicative only, is subject to
> change and does not constitute an offer to deal at any price quoted. Any
> reference to the terms of executed transactions should be treated as
> preliminary only and subject to our formal written confirmation.
>