You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by David Thomas <dt...@gmail.com> on 2014/03/11 07:06:06 UTC

Block

What is the concept of Block and BlockManager in Spark? How is a Block
related to a Partition of a RDD?

Re: Block

Posted by Patrick Wendell <pw...@gmail.com>.

A block is an internal construct that isn't directly exposed to users.
Internally though, each partition of an RDD is mapped to one block.

- Patrick

On Mon, Mar 10, 2014 at 11:06 PM, David Thomas <dt...@gmail.com> wrote:
> What is the concept of Block and BlockManager in Spark? How is a Block
> related to a Partition of a RDD?

Re: Block

Posted by dachuan <hd...@gmail.com>.

In my opinion, BlockManager manages many types of Block, RDD's partition,
a.k.a. RDDBlock, is one type of them. Other types of Blocks are
ShuffleBlock, IndirectBlock (if the task's return status is too large), etc.

So, BlockManager is a layer that is independent of RDD concept.
On Mar 11, 2014 2:06 AM, "David Thomas" <dt...@gmail.com> wrote:

> What is the concept of Block and BlockManager in Spark? How is a Block
> related to a Partition of a RDD?
>