[jira] [Created] (ARROW-4294) [Plasma] Add support for evicting objects to external store

["Anurag Khandelwal (JIRA)" <ji...@apache.org>]

Anurag Khandelwal created ARROW-4294:
----------------------------------------

             Summary: [Plasma] Add support for evicting objects to external store
                 Key: ARROW-4294
                 URL: https://issues.apache.org/jira/browse/ARROW-4294
             Project: Apache Arrow
          Issue Type: New Feature
          Components: C++
    Affects Versions: 0.11.1
            Reporter: Anurag Khandelwal
             Fix For: 0.13.0


Currently, when Plasma needs storage space for additional objects, it evicts objects by deleting them from the Plasma store. This is a problem when it isn't possible to reconstruct the object or reconstructing it is expensive. Adding support for a pluggable external store that Plasma can evict objects to will address this issue. 

My proposal is described below.

*Requirements*
 * Objects in Plasma should be evicted to a external store rather than being removed altogether
 * Communication to the external storage service should be through a very thin, shim interface. At the same time, the interface should be general enough to support arbitrary remote services (e.g., S3, DynamoDB, Redis, etc.)
 * Should be pluggable (e.g., it should be simple to add in or remove the external storage service for eviction, switch between different remote services, etc.) and easy to implement

*Assumptions/Non-Requirements*
 * The external store has practically infinite storage
 * The external store's write operation is idempotent and atomic; this is needed ensure there are no race conditions due to multiple concurrent evictions of the same object.

*Proposed Implementation*
 * Define a ExternalStore interface with a Connect call. The call returns an ExternalStoreHandle, that exposes Put and Get calls. Any external store that needs to be supported has to have this interface implemented.
 * In order to read or write data to the external store in a thread-safe manner, one ExternalStoreHandle should be created per-thread. While the ExternalStoreHandle itself is not required to be thread-safe, multiple ExternalStoreHandles across multiple threads should be able to modify the external store in a thread-safe manner.
 * Replace the DeleteObjects method in the Plasma Store with an EvictObjects method. If an external store is specified for the Plasma store, the EvictObjects method would mark the object state as PLASMA_EVICTED, write the object data to the external store (via the ExternalStoreHandle) and reclaim the memory associated with the object data/metadata rather than remove the entry from the Object Table altogether. In case there is no valid external store, the eviction path would remain the same (i.e., the object entry is still deleted from the Object Table).
 * The Get method in Plasma Store now tries to fetch the object from external store if it is not found locally and there is an external store associated with the Plasma Store. The method tries to offload this to an external worker thread pool with a fire-and-forget model, but may need to do this synchronously if there are too many requests already enqueued.
 * *The CMake build system can expose a variable, EXTERNAL_STORE_SOURCES, which can be appended to with implementations of the ExternalStore and ExternalStoreHandle interfaces, which will then be compiled into the plasma_store_server executable.*

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)