[jira] [Commented] (TRAFODION-1500) ESP co-location needs to consider some nodes may host too many ESP processes due to data in-balance

["Qifan Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TRAFODION-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908140#comment-14908140 ] 

Qifan Chen commented on TRAFODION-1500:
---------------------------------------

Smooth the node map to reassign entries with identical node Id of higher
frequency than the rest to some other nodes. 

Assume there are m total entries in the map, and 
n nodes in the cluster. Assume further there are s entries in the map (out of m) that refer 
to very few nodes. These s entrres are the subject of smoothing operation.  We do so by

1. allocate an array nodeUsageMap[] and the ith entry in it contains all indexes k in the map
    that points at i (i.e., nodeMap.getNodeId[k] = i)
2. Find out s by frequency counting
3. Find out m - s
4. Find out how many nodes in each of the s entries that should be moved
 
The nodels accepting the reasssignment will be from the set of the nodes contained in the map.

// Example .  Assume the original node map with 6 entries as follows:
//
//    NodeMapIndex   0   1   2   3  4  5 
//    NodeMapEntry   0   1   1   3  1  5
//
// The nodes accepting the reasssignment will be { 0, 3, 5 }
//
// The subset of entries referring to a few nodes with high frequency: s = { 1, 2, 4 } 
// Since f(0)=f(3)=f(5)=1, the average frequency of nodes not in s: (1+1+1)/3 = 1.
// We will allow one (1) assigment in s to remain in node 1 since it is the average frequency, and 
// re-assign the rest (marked X) to different nodes via round-robin starting the 1st node in the map. 
// The node (1) is excluded from the re-assignment. 
//
//    NodeMapIndex   0   1   2   3  4  5 
//    NodeMapEntry   0   1   1   3  1  5
//                                   X    X
//                                  ^      ^
//                                  |      |
//                               to 0   to 3
//
// The final smoothed node map:
//
//    NodeMapIndex   0   1   2   3  4  5 
//    NodeMapEntry   0   1   0   3  3  5

> ESP co-location needs to consider some nodes may host too many ESP processes due to data in-balance
> ---------------------------------------------------------------------------------------------------
>
>                 Key: TRAFODION-1500
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-1500
>             Project: Apache Trafodion
>          Issue Type: Bug
>          Components: sql-cmp
>            Reporter: Qifan Chen
>
> When a table is range-partitioned, or stats partitioned, it is possible that the amount of the data hosted at each region server may not be uniform.  Assume region server s running on node n hosts more data than the rest of the region servers. With the current ESP co-location assignment,  node s will be assigned with more # of ESPs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)