You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "xumanbu (via GitHub)" <gi...@apache.org> on 2026/01/13 03:18:26 UTC
[I] [Qualification]Operator Count Inflation in UnsupportedOperators Report [incubator-gluten]
xumanbu opened a new issue, #11394:
URL: https://github.com/apache/incubator-gluten/issues/11394
### Backend
VL (Velox)
### Bug description
### Problem
The `UnsupportedOperators.tsv` report shows inflated operator counts that don't match the actual number of operator instances in the execution plan.
### Example
For this execution plan:
```
Execute InsertIntoHadoopFsRelationCommand (node 0, unsupported #1 - ROOT)
└── Project (node 1)
└── SortMergeJoin(skew=true) (node 2, unsupported #2)
├── Scan table1 (node 3)
└── Scan table2 (node 4)
```
**Expected**: 2 operators (1 InsertIntoHadoopFsRelationCommand + 1 SortMergeJoin)
**Actual**: 4 in the report
In production event logs, the issue is more severe:
- `Execute InsertIntoHadoopFsRelationCommand`: reported 62, should be 4
- `SortMergeJoin(skew=true)`: reported 62, should be 4
### Root Cause
In `ResultVisitor.java`, the `increment()` method is called for every parent node during post-order traversal:
```java
// Current (wrong) behavior:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
currentCost.increment(); // ❌ Called for each parent node
});
```
When traversing node 1 (Project), it inherits unsupported operator from node 2 (SortMergeJoin) and increments the count again. Same for node 0, causing the count to be 3 instead of 1 for the SortMergeJoin.
### Solution
Only count when the operator is first encountered:
```java
// Fixed behavior:
if (isOperatorNotSupported(nodeId)) {
currentNodeUnsupportedOperators.add(nodeId);
unsupportedOperatorImpactCostMap.put(nodeId, new UnsupportedImpact());
unsupportedOperatorImpactCostMap.get(nodeId).increment(); // ✅ Count once
}
// When visiting parent nodes, only accumulate CPU time:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
// No increment() here
});
```
### Impact
- Count values are 3-15x higher than actual operator instances
- Makes the report misleading for users
- CPU time metrics are NOT affected (still correct)
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [Qualification]Operator Count Inflation in UnsupportedOperators Report [incubator-gluten]
Posted by "luomh1998 (via GitHub)" <gi...@apache.org>.
luomh1998 commented on issue #11394:
URL: https://github.com/apache/incubator-gluten/issues/11394#issuecomment-3744613941
i have a question. In your example, why expect Execute InsertIntoHadoopFsRelationCommand: should be 4 instead of 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org