[jira] [Commented] (DRILL-1414) Move profile storage to DFS rather than using PStore

["Aditya Kishore (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147194#comment-14147194 ] 

Aditya Kishore commented on DRILL-1414:
---------------------------------------

So I have been thinking about couple of ways to do it.

# Extend {{org.apache.drill.exec.store.sys.PStore}} interface to add two additional functions
{code}
  public V getBlob(String key);
  public void putBlob(String key, V value);
{code}
Now these two methods can be used by the consumers to store large amount of data, that may not require frequent enumeration and not suitable for storage on systems like Zookeeper. A particular PStore implementation could choose to store the blob data differently than the primary value, for example, HBase PStore provider could store them in a different column family while Zookeeper PStore provider can store them on DFS (as this JIRA summary suggests).
The Query Profile, then can be split into two part where small, meta info about the query is stored with a {{put()}} while the fragment profiles are stored using {{putBlob()}}.
# Alternatively, we could handle this narrowly by just modifying {{org.apache.drill.exec.work.foreman.QueryStatus}} to split and store the profile meta data separately form individual query profile.

I am inclined to go with approach #1 as it will allow any future consumer to reuse it effortlessly. I already have a partial patch, excluding modification to the Web UI, that I am currently testing at this moment. If I do not hear any concern with the approach #1, I'll post the patch shortly for the review.

> Move profile storage to DFS rather than using PStore
> ----------------------------------------------------
>
>                 Key: DRILL-1414
>                 URL: https://issues.apache.org/jira/browse/DRILL-1414
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Jacques Nadeau
>            Assignee: Aditya Kishore
>             Fix For: 0.6.0
>
>
> PStores were really built for trivial configuration data, not large query profiles.  As such, we should move to using the DFS for storage of query profiles when distributed mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)