[jira] [Comment Edited] (REEF-580) Add a Block Management Service to REEF

["Andrew Chung (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009554#comment-15009554 ] 

Andrew Chung edited comment on REEF-580 at 11/17/15 10:16 PM:
--------------------------------------------------------------

Here's an initial plan to tackle the feature:

# Describe and document overview/guarantees of Blocks, Partitions, and DataSets. What do we want to support? What are the overall capabilities of each? Ordering of blocks, partitions, and datasets.
# Design an interface for IBlock and related interfaces IPartition, IDataSet, and check it in.
# Implement the different types of IBlocks – Local, Disk, Remote, External. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 9.
# Design the interface of BlockManager.
# Implement the basic in-memory operations of BlockManager, such as GetBlock, GetBlocksByPartition, GetBlocksByDataSet…etc, and check it in. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 10 and 11.
# Implement the basic on-disk operations of BlockManager, such as WriteBlockToDisk, ReadBlockFromDisk…etc. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 10 and 11.
# Design and implement a BlockReference for the BlockManagerMaster that reports where an associated block is living.
# Design the interface of and implement the basic operations of BlockManagerMaster, such as GetBlockReferences, GetBlockReferencesByPartition…etc, and check it in. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagers, which will ultimately be solved in step 10.
# Design and implement heartbeats between BlockManagerMaster and BlockManagers such that BlockManagers send a list of BlockReferences periodically to the BlockManagerMaster.
# Wire together BlockManagerMaster and BlockManagers, which will allow BlockManagers to get BlockReferences from BlockManagerMaster.
# Add communication channel between BlockManagers for remote block fetching.
# Additional testing. E2E testing.
# Adding frameworks on top of BlockManager!

For the ordering of sub-tasks:
1 blocks 2, and 2 blocks everything. 3 is parallelizable for each implementation. The in-memory portion of 3 blocks 5, and the disk portion of 3 blocks 6. 4 and 7 are parallelizable, while 4 blocks 5 and 6. 7 blocks 8. 5, (maybe) 6, and 8 block 9 and 11. 9 blocks 10, but 9 and 11 are parallelizable.

How does this sound? If this looks good, I'll begin creating sub-tasks.


was (Author: afchung90):
Here's an initial plan to tackle the feature:

1. Describe and document overview/guarantees of Blocks, Partitions, and DataSets. What do we want to support? What are the overall capabilities of each? Ordering of blocks, partitions, and datasets.
2. Design an interface for IBlock and related interfaces IPartition, IDataSet, and check it in.
3. Implement the different types of IBlocks – Local, Disk, Remote, External. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 9.
4. Design the interface of BlockManager.
5. Implement the basic in-memory operations of BlockManager, such as GetBlock, GetBlocksByPartition, GetBlocksByDataSet…etc, and check it in. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 10 and 11.
6. Implement the basic on-disk operations of BlockManager, such as WriteBlockToDisk, ReadBlockFromDisk…etc. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagerMaster and/or other BlockManagers, which will ultimately be solved in step 10 and 11.
7. Design and implement a BlockReference for the BlockManagerMaster that reports where an associated block is living.
8. Design the interface of and implement the basic operations of BlockManagerMaster, such as GetBlockReferences, GetBlockReferencesByPartition…etc, and check it in. Throw NotImplementedExceptions and add JIRA TODOs for methods that require communication with BlockManagers, which will ultimately be solved in step 10.
9. Design and implement heartbeats between BlockManagerMaster and BlockManagers such that BlockManagers send a list of BlockReferences periodically to the BlockManagerMaster.
10. Wire together BlockManagerMaster and BlockManagers, which will allow BlockManagers to get BlockReferences from BlockManagerMaster.
11. Add communication channel between BlockManagers for remote block fetching.
12. Additional testing. E2E testing.
13. Adding frameworks on top of BlockManager!

For the ordering of sub-tasks:
1 blocks 2, and 2 blocks everything. 3 is parallelizable for each implementation. The in-memory portion of 3 blocks 5, and the disk portion of 3 blocks 6. 4 and 7 are parallelizable, while 4 blocks 5 and 6. 7 blocks 8. 5, (maybe) 6, and 8 block 9 and 11. 9 blocks 10, but 9 and 11 are parallelizable.

How does this sound? If this looks good, I'll begin creating sub-tasks.

> Add a Block Management Service to REEF
> --------------------------------------
>
>                 Key: REEF-580
>                 URL: https://issues.apache.org/jira/browse/REEF-580
>             Project: REEF
>          Issue Type: New Feature
>            Reporter: Markus Weimer
>
> We propose the addition of a data Block Management service to REEF. The Block Manager manages the transient data of a Big Data application. The Block Manager assumes that transient data can be managed in the following hierarchy:
>   * *Data Set:* A data set consists of a set of (physical)n partitions. For instance, a folder on HDFS could be considered a data set, while its files constitute the partitions.
>   * *Partition:* a physical partition of a data set. In the example above, it would be a file. Partitions consist of Blocks.
>   * *Block:* The atomic unit of data management. Each block belongs to exactly one partition. Blocks are immutable. Blocks can be stored in Evaluator memory, on local Disk or stable, distributed storage. Blocks can have replicas across these memory tiers. Blocks contain data of arbitrary format. From the perspective of this Block Management service, they are large, fixed sized byte arrays.
> The purpose of the Block Manager is to manage the metadata and movement of data sets organized in such a way. To facilitate that, each Block, Partition and DataSet has a unique ID.
> On the *Task side*, the Block Manager facilitates the retrieval of and access to any Block or Partition by their ID. Specific access methods are yet to be designed (e.g. whether or not there is an order to the blocks). Also, new Blocks can be created on the Task side for a given Partition. Special consideration shall be given to the memory allocation efficiency of this operation.
> On the *Driver side*, the Block Manager keeps track of the metadata of all Blocks. It provides a network protocol used by the Task side components to retrieve and update metadata records. Metadata can be kept in memory or, in a later version, in stable storage such as a SQL database.
> The Block Management service shall be built in a language and platform agnostic manner. At the very least, the Driver side network protocol needs to be accessible by both JVM and CLR implementations of the Task side. REST could be an appropriate approach.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)