[jira] [Created] (DRILL-1990) Add peak memory allocation in a operator to OperatorStats.

["Venki Korukanti (JIRA)" <ji...@apache.org>]

Venki Korukanti created DRILL-1990:
--------------------------------------

             Summary: Add peak memory allocation in a operator to OperatorStats.
                 Key: DRILL-1990
                 URL: https://issues.apache.org/jira/browse/DRILL-1990
             Project: Apache Drill
          Issue Type: Improvement
          Components: Execution - Operators
            Reporter: Venki Korukanti
            Assignee: Venki Korukanti
             Fix For: 0.8.0


Currently we have "localMemoryAllocated" which is always set to zero as the we try to fill the stats at the end of execution of fragment by calling calling allocator. getAllocatedMemory() which at that point has already released the allocated memory. Instead having a peak memory that allocator (each operator has its own allocator) has seen in the lifetime of the operator execution will be useful.

Example query on query profile: To find all aggregate peak memory of each operator across all minor fragments in a major fragment and list them in descending order
{code:sql}
SELECT
  majorFragmentId,
  opProfile['operatorType'] opType,
  sum(opProfile['peakLocalMemoryAllocated']) aggPeakMemoryAcrossAllMinorFragments 
FROM 
  (SELECT
       majorFragmentId,
       flatten(minorFragProfile['operatorProfile']) opProfile
   FROM 
       (SELECT
             majorFragment['majorFragmentId'] majorFragmentId, 
             flatten(majorFragment['minorFragmentProfile']) minorFragProfile
         FROM
             (SELECT flatten(fragmentProfile) as majorFragment from dfs.`/tmp/a.json`)
         )
   )
-- WHERE opProfile['operatorType'] = 6 -- If want to filter to particular operator
GROUP BY 
  majorFragmentId,
  opProfile['operatorType']
ORDER BY
  aggPeakMemoryAcrossAllMinorFragments DESC;
{code}

{code}
+-----------------+------------+--------------------------------------+
| majorFragmentId |   opType   | aggPeakMemoryAcrossAllMinorFragments |
+-----------------+------------+--------------------------------------+
| 1               | 4          | 115065856                            |
| 1               | 3          | 10027008                             |
| 0               | 3          | 1671168                              |
| 3               | 6          | 1536000                              |
| 2               | 6          | 901120                               |
| 1               | 6          | 606208                               |
| 3               | 28         | 393216                               |
| 2               | 28         | 229376                               |
| 3               | 10         | 122880                               |
| 2               | 10         | 81920                                |
| 0               | 11         | 0                                    |
| 0               | 10         | 0                                    |
| 0               | 13         | 0                                    |
| 1               | 10         | 0                                    |
| 1               | 11         | 0                                    |
+-----------------+------------+--------------------------------------+
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)