You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "xumanbu (via GitHub)" <gi...@apache.org> on 2026/01/13 03:18:26 UTC

[I] [Qualification]Operator Count Inflation in UnsupportedOperators Report [incubator-gluten]

xumanbu opened a new issue, #11394:
URL: https://github.com/apache/incubator-gluten/issues/11394

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   ### Problem
   
   The `UnsupportedOperators.tsv` report shows inflated operator counts that don't match the actual number of operator instances in the execution plan.
   
   ### Example
   
   For this execution plan:
   ```
   Execute InsertIntoHadoopFsRelationCommand (node 0, unsupported #1 - ROOT)
     └── Project (node 1)
         └── SortMergeJoin(skew=true) (node 2, unsupported #2)
             ├── Scan table1 (node 3)
             └── Scan table2 (node 4)
   ```
   
   **Expected**: 2 operators (1 InsertIntoHadoopFsRelationCommand + 1 SortMergeJoin)
   **Actual**: 4 in the report
   
   In production event logs, the issue is more severe:
   - `Execute InsertIntoHadoopFsRelationCommand`: reported 62, should be 4
   - `SortMergeJoin(skew=true)`: reported 62, should be 4
   
   ### Root Cause
   
   In `ResultVisitor.java`, the `increment()` method is called for every parent node during post-order traversal:
   
   ```java
   // Current (wrong) behavior:
   currentNodeUnsupportedOperators.forEach(id -> {
       currentCost.addCpuDuration(durationMetric.getDuration());
       currentCost.increment();  // ❌ Called for each parent node
   });
   ```
   
   When traversing node 1 (Project), it inherits unsupported operator from node 2 (SortMergeJoin) and increments the count again. Same for node 0, causing the count to be 3 instead of 1 for the SortMergeJoin.
   
   ### Solution
   
   Only count when the operator is first encountered:
   
   ```java
   // Fixed behavior:
   if (isOperatorNotSupported(nodeId)) {
       currentNodeUnsupportedOperators.add(nodeId);
       unsupportedOperatorImpactCostMap.put(nodeId, new UnsupportedImpact());
       unsupportedOperatorImpactCostMap.get(nodeId).increment();  // ✅ Count once
   }
   
   // When visiting parent nodes, only accumulate CPU time:
   currentNodeUnsupportedOperators.forEach(id -> {
       currentCost.addCpuDuration(durationMetric.getDuration());
       // No increment() here
   });
   ```
   
   ### Impact
   
   - Count values are 3-15x higher than actual operator instances
   - Makes the report misleading for users
   - CPU time metrics are NOT affected (still correct)
   
   
   ### Gluten version
   
   _No response_
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   ```bash
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [Qualification]Operator Count Inflation in UnsupportedOperators Report [incubator-gluten]

Posted by "luomh1998 (via GitHub)" <gi...@apache.org>.
luomh1998 commented on issue #11394:
URL: https://github.com/apache/incubator-gluten/issues/11394#issuecomment-3744613941

   i have a question. In your example, why expect Execute InsertIntoHadoopFsRelationCommand: should be 4 instead of 1?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org