[jira] [Commented] (IMPALA-4080) Share codegen work between fragment instances

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073239#comment-17073239 ] 

ASF subversion and git services commented on IMPALA-4080:
---------------------------------------------------------

Commit f2837e97b25be37221ab475b0fd4f66320a4f66b in impala's branch refs/heads/master from Bikramjeet Vig
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f2837e9 ]

IMPALA-4080 [part 7]: Codegen once per fragment

This is the final patch:
- Removes all temporary changes that called codegen on plan nodes and
  data sink through the exec node.
- Introduces FragmentState class that basically contains the plan tree,
  data sink config and the codegen state for every fragment.
- Every FragmentState object has its own runtime profile that helps to
  map out every codegen object's profile to its fragment id.
- Plan nodes are generated before fragment instance threads are spawned.
- The first fragment instance thread to call codegen on its parent
  parent fragment does the actual codegen work and the rest of the
  instance threads wait till codegen is complete.
- This very closely mimics the timing of the state machines both for
  query state and the fragment instance states.

Testing:
- Ran exhaustive tests successfully.

An example of how the codegen in the profile looks:

Per Node Profiles:
  localhost:22002:
     - AdmissionSlots: 1 (1)
     - BloomFilterBytes: 16.00 MB (16777216)
     - ScratchBytesRead: 0
     - ScratchBytesWritten: 0
     - ScratchFileUsedBytes: 0
     - ScratchReads: 0 (0)
     - ScratchWrites: 0 (0)
     - TotalEncryptionTime: 0.000ns
     - TotalReadBlockTime: 0.000ns
    Buffer pool:
       - AllocTime: 153.663us
       - CumulativeAllocationBytes: 16.00 MB (16777216)
       - CumulativeAllocations: 8 (8)
       - PeakReservation: 16.00 MB (16777216)
       - PeakUnpinnedBytes: 0
       - PeakUsedReservation: 16.00 MB (16777216)
       - ReadIoBytes: 0
       - ReadIoOps: 0 (0)
       - ReadIoWaitTime: 0.000ns
       - ReservationLimit: 16.00 MB (16777216)
       - SystemAllocTime: 133.841us
       - WriteIoBytes: 0
       - WriteIoOps: 0 (0)
       - WriteIoWaitTime: 0.000ns
    Fragment F02:
      CodeGen:(Total: 695.207ms, non-child: 0.000ns, % non-child: 0.00%)
         - CodegenInvoluntaryContextSwitches: 102 (102)
         - CodegenTotalWallClockTime: 695.180ms
           - CodegenSysTime: 0.000ns
           - CodegenUserTime: 688.998ms
         - CodegenVoluntaryContextSwitches: 22 (22)
         - CompileTime: 39.657ms
         - IrGenerationTime: 27.443ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.53 MB (2647876)
         - NumFunctions: 161 (161)
         - NumInstructions: 7.13K (7128)
         - OptimizationTime: 596.075ms
         - PeakMemoryUsage: 3.48 MB (3649536)
         - PrepareTime: 30.782ms
    Fragment F00:
      CodeGen:(Total: 145.314ms, non-child: 0.000ns, % non-child: 0.00%)
         - CodegenInvoluntaryContextSwitches: 11 (11)
         - CodegenTotalWallClockTime: 145.304ms
           - CodegenSysTime: 3.944ms
           - CodegenUserTime: 137.738ms
         - CodegenVoluntaryContextSwitches: 4 (4)
         - CompileTime: 6.697ms
         - IrGenerationTime: 3.173ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.53 MB (2647876)
         - NumFunctions: 41 (41)
         - NumInstructions: 1.02K (1024)
         - OptimizationTime: 119.623ms
         - PeakMemoryUsage: 512.00 KB (524288)
         - PrepareTime: 14.405ms

Change-Id: I3aef8bc621f96caafe9a1c378617a2987e4ad452
Reviewed-on: http://gerrit.cloudera.org:8080/15408
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Share codegen work between fragment instances
> ---------------------------------------------
>
>                 Key: IMPALA-4080
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4080
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.8.0
>            Reporter: Tim Armstrong
>            Assignee: Bikramjeet Vig
>            Priority: Major
>              Labels: codegen, multithreading
>
> With multithreading, we should find a way to share codegen modules between fragment instances to avoid redundant compilation, optimisation, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org