You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "SubhamSinghal (via GitHub)" <gi...@apache.org> on 2026/09/05 14:12:53 UTC
[PR] Pwmj metrics accounting [datafusion]
SubhamSinghal opened a new pull request, #24956:
URL: https://github.com/apache/datafusion/pull/24956
## Which issue does this PR close?
Part of #17427.
## Rationale for this change
`PiecewiseMergeJoinExec`'s metrics accounting had several gaps left over from earlier PRs in the epic:
- The classic-join stream (`Left`/`Right`/`Full`/`Inner`) never routed `poll_next` through `BaselineMetrics::record_poll`, so `output_rows`, `output_bytes`, and `output_batches` stayed at `0` in `EXPLAIN ANALYZE` regardless of how many rows the join actually produced.
- `join_time` was never measured on either the classic or existence-join stream — only `build_time` was timed, so `elapsed_compute` understated total operator cost for large probe sides.
- `probe_hit_rate`/`avg_fanout` exist on the shared `BuildProbeJoinMetrics` struct (populated by `HashJoinExec`) but PWMJ never populated them, always showing `N/A (0/0)`.
## What changes are included in this PR?
- `ClassicPWMJStream::poll_next` now calls `self.join_metrics.baseline.record_poll(poll)`, matching the existence-join stream.
- `join_time` is timed around the actual comparison work: `resolve_classic_join` and the `ProcessUnmatched` bitmap/take pass on the classic path; `extreme_key` + `mark_matched_buffered_rows` on the existence path.
- `probe_hit_rate`/`avg_fanout` are populated for classic join, using the range size (`buffered_len - buffer_idx`) already computed at each match as the fanout.
- `probe_hit_rate` is populated for existence join: a streamed batch counts as a hit if its extreme key lowers the shared watermark, a miss otherwise. `avg_fanout` is intentionally left unset for existence join — its watermark-based semantics don't have a natural per-row fanout equivalent.
- Added a metrics-focused test to each stream module: `classic_join::tests::inner_join_records_output_and_probe_metrics` and `existence_join::tests::probe_hit_rate_counts_batches_that_advance_the_watermark`, both hand-derived and verified against actual runs.
## Are these changes tested?
Yes — two new unit tests
## Are there any user-facing changes?
`EXPLAIN ANALYZE` on a `PiecewiseMergeJoinExec` plan now reports accurate `output_rows`/`output_bytes`/`output_batches`/`join_time`/`probe_hit_rate`/`avg_fanout` instead of zeros/`N/A`. No API or behavior change to query results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#issuecomment-5557844772
I found a possible issue
`probe_hit_rate` for existence join is defined as "lowered the watermark", which isn't a hit rate. The new test shows it: buffered 1..5, `Gt`, streamed key 4 matches buffered 5, but it's asserted as a miss because key 3 arrived first. I ran the same test with the two batches swapped and got 2/2 instead of 1/2. With multiple streamed partitions, whichever partition lowers `min_marked` first makes every later partition's matches look like misses, and once the watermark reaches `first_non_null_buffered` the whole block is skipped so every remaining batch is a miss. `docs/source/user-guide/metrics.md` defines this as "fraction of probe-side rows with a build-side join-key match", so an `EXPLAIN ANALYZE` reader will take 50% to mean half the probe side didn't match.
Since `is_match` is monotone over the sorted buffered side, "does this batch match anything" is one comparison against the last buffered row, independent of the watermark. Decide the hit there, then keep the bounded binary search only for the watermark update:
```rust
if row_idx < stream_values.len() && first_non_null_buffered < buffered_len {
let cmp = JoinKeyComparator::new(/* unchanged */)?;
let is_match = |buffer_idx: usize| { /* unchanged */ };
// `is_match` is monotone over the sorted buffered side, so the extreme key
// matches *something* iff it matches the last buffered key. That decides
// `probe_hit_rate` independently of the watermark, which other batches or
// partitions may already have lowered past this batch's match range. A
// batch that matches nothing can't lower the watermark either.
if !is_match(buffered_len - 1) {
return Ok(());
}
self.join_metrics.probe_hit_rate.add_part(1);
if first_non_null_buffered >= scan_limit {
// Everything this batch could mark is already marked.
return Ok(());
}
// existing binary search over [first_non_null_buffered, scan_limit) ...
let buffer_idx = lo;
if buffer_idx < scan_limit {
buffered_data.min_marked.fetch_min(buffer_idx, AtomicOrdering::SeqCst);
}
}
```
The existing test then expects `(2, 2)` since key 4 really does match buffered 5. To guard the order dependence, a test that runs the same batches in both orders, with one batch that genuinely matches nothing (key 5 under `>`) and one pair of nested hits:
```rust
/// Runs a `LeftSemi` `buffered.b1 > streamed.b1` join over buffered keys 1..=5 with
/// the given streamed batches in one partition and returns `probe_hit_rate` as
/// `(part, total)`.
async fn existence_probe_hit_rate(
streamed_batches: Vec<RecordBatch>,
) -> Result<(usize, usize)> {
let left = build_table(
("a1", &vec![1, 2, 3, 4, 5]),
("b1", &vec![1, 2, 3, 4, 5]),
("c1", &vec![10, 20, 30, 40, 50]),
);
let streamed_schema = Schema::new(vec![
Field::new("a2", DataType::Int32, false),
Field::new("b1", DataType::Int32, false),
Field::new("c2", DataType::Int32, false),
]);
let right = TestMemoryExec::try_new_exec(
&[streamed_batches],
Arc::new(streamed_schema),
None,
)?;
let on = (
Arc::new(Column::new_with_schema("b1", &left.schema())?) as _,
Arc::new(Column::new_with_schema("b1", &right.schema())?) as _,
);
let join = PiecewiseMergeJoinExec::try_new(
left, right, on, Operator::Gt, JoinType::LeftSemi, 1,
)?;
let stream = join.execute(0, Arc::new(TaskContext::default()))?;
common::collect(stream).await?;
let metrics = join.metrics().unwrap();
Ok(metrics
.iter()
.find_map(|m| match m.value() {
crate::metrics::MetricValue::Ratio { name, ratio_metrics }
if name == "probe_hit_rate" =>
{
Some((ratio_metrics.part(), ratio_metrics.total()))
}
_ => None,
})
.expect("probe_hit_rate metric"))
}
/// `probe_hit_rate` must describe the data, not the order it arrived in. Key 3 matches
/// buffered 4 and 5; key 5 matches nothing under `>`. Whichever batch lowers the
/// watermark first, the answer is one hit out of two.
#[tokio::test]
async fn existence_probe_hit_rate_is_independent_of_batch_order() -> Result<()> {
let hit = || build_table_i32(("a2", &vec![10]), ("b1", &vec![3]), ("c2", &vec![70]));
let miss = || build_table_i32(("a2", &vec![20]), ("b1", &vec![5]), ("c2", &vec![80]));
assert_eq!(existence_probe_hit_rate(vec![hit(), miss()]).await?, (1, 2));
assert_eq!(existence_probe_hit_rate(vec![miss(), hit()]).await?, (1, 2));
// Two hits whose match ranges nest: the second cannot lower the watermark but
// still matched, so it must not be reported as a miss in either order.
let inner = || build_table_i32(("a2", &vec![30]), ("b1", &vec![4]), ("c2", &vec![90]));
assert_eq!(existence_probe_hit_rate(vec![hit(), inner()]).await?, (2, 2));
assert_eq!(existence_probe_hit_rate(vec![inner(), hit()]).await?, (2, 2));
Ok(())
}
```
On the current branch this test fails at the nested-hits assertion with `(1, 2)`; with the change above it passes along with the rest of the module and clippy. If you'd rather not add the compare, leaving the metric unset (as you did for `avg_fanout`) is better than a number that changes with batch order. Either way the metrics doc table should say PWMJ existence joins count streamed batches rather than rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#issuecomment-5552534645
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24956?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `88.31776%` with `25 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.61%. Comparing base ([`35f58f5`](https://app.codecov.io/gh/apache/datafusion/commit/35f58f53cde2f634c21f6370e385cf1d7e9bc55c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d24e66a`](https://app.codecov.io/gh/apache/datafusion/commit/d24e66a93ff0d60806280f2827b2e75c22ab4938?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24956?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...n/src/joins/piecewise\_merge\_join/existence\_join.rs](https://app.codecov.io/gh/apache/datafusion/pull/24956?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fpiecewise_merge_join%2Fexistence_join.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9waWVjZXdpc2VfbWVyZ2Vfam9pbi9leGlzdGVuY2Vfam9pbi5ycw==) | 88.03% | [4 Missing and 10 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...lan/src/joins/piecewise\_merge\_join/classic\_join.rs](https://app.codecov.io/gh/apache/datafusion/pull/24956?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fpiecewise_merge_join%2Fclassic_join.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9waWVjZXdpc2VfbWVyZ2Vfam9pbi9jbGFzc2ljX2pvaW4ucnM=) | 88.65% | [0 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24956 +/- ##
========================================
Coverage 81.61% 81.61%
========================================
Files 1124 1124
Lines 411978 412288 +310
Branches 411978 412288 +310
========================================
+ Hits 336236 336499 +263
- Misses 55936 55955 +19
- Partials 19806 19834 +28
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24956?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "SubhamSinghal (via GitHub)" <gi...@apache.org>.
SubhamSinghal commented on code in PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#discussion_r3943078005
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -336,6 +340,7 @@ impl ClassicPWMJStream {
let buffered_data = Arc::clone(&self.buffered_side.try_as_ready()?.buffered_data);
let buffered_batch = buffered_data.batch();
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
Addressed in a0ba9a7897a11f178e009921a5fbb101492ab061
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/existence_join.rs:
##########
@@ -241,12 +241,19 @@ impl ExistencePWMJStream {
self.join_metrics.input_batches.add(1);
self.join_metrics.input_rows.add(batch.num_rows());
- // Only the batch's extreme key is ever compared against the buffered side,
- // so reduce the batch to that one key.
- let stream_values =
- extreme_key(&stream_values, self.sort_option.descending)?;
-
- self.mark_matched_buffered_rows(&stream_values)?;
+ // An empty batch has no extreme key to compare, so it can neither match
+ // nor miss -- counting it either way would understate the real hit rate.
+ if batch.num_rows() > 0 {
+ let join_time = self.join_metrics.join_time.clone();
+ let join_timer = join_time.timer();
Review Comment:
Addressed in https://github.com/apache/datafusion/commit/a0ba9a7897a11f178e009921a5fbb101492ab061
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -294,6 +295,7 @@ impl ClassicPWMJStream {
}
// Produce more work
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
Addressed in https://github.com/apache/datafusion/commit/a0ba9a7897a11f178e009921a5fbb101492ab061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#issuecomment-5603695263
@SubhamSinghal I still have a suggestion for the latest commit,
**`probe_hit_rate` is per-row everywhere else — the existence path publishes batches under that name.**
`existence_join.rs:301` does `add_total(1)` per streamed batch and `:360` does `add_part(1)` per matching batch. But the metric is defined as *rows*:
- `joins/utils.rs:1843` — "Fraction of probe rows with at least one build-side join-key match", under a `// Keep these metric descriptions in sync with docs/source/user-guide/metrics.md` note
- `docs/source/user-guide/metrics.md:60` — same wording
- `classic_join.rs:245` (this PR) — `probe_hit_rate.add_total(batch.num_rows())`
- `hash_join/stream.rs:765` — `probe_hit_rate.add_total(state.batch.num_rows())`
So a `LeftSemi` PWMJ over two 8192-row batches in which 3 rows actually match renders `probe_hit_rate: 100% (2/2)`. That doesn't just lose resolution, it reports the opposite of the truth, and `probe_hit_rate` now means two different things in two streams of the same operator. It also breaks the invariant the other two paths hold: `probe_hit_rate.total == input_rows`.
The good news is the fix is cheap. The predicate you already evaluate for the hit decision — `is_match(buffered_len - 1)`, i.e. "does this key beat the most-matching buffered key" — is exactly the per-row predicate, and it vectorizes, so the O(log n) watermark search is untouched:
```rust
// scan_stream_batch, before reducing to the extreme key
if batch.num_rows() > 0 {
self.record_probe_hits(&stream_values)?;
let extreme = extreme_key(&stream_values, self.sort_option.descending)?;
self.mark_matched_buffered_rows(&extreme)?;
}
```
```rust
/// `probe_hit_rate` is defined per probe *row*. A streamed row matches iff it satisfies
/// the predicate against the most-matching buffered key, which is the last non-null
/// buffered row (`is_match` is monotone over the sorted buffered side) -- so one
/// vectorized comparison per batch is enough.
fn record_probe_hits(&self, stream_values: &ArrayRef) -> Result<()> {
self.join_metrics.probe_hit_rate.add_total(stream_values.len());
let buffered_values = self.buffered_side.try_as_ready()?.buffered_data.values();
let len = buffered_values.len();
if len == buffered_values.null_count() {
return Ok(()); // no non-null buffered key: nothing can match
}
let last = Scalar::new(buffered_values.slice(len - 1, 1));
// The join predicate is `buffered OP streamed`, so flip it to compare
// streamed against the key. Null streamed keys compare to null and are
// excluded by `true_count()`, which is what we want.
let hits = match self.operator {
Operator::Gt => lt(stream_values, &last)?,
Operator::GtEq => lt_eq(stream_values, &last)?,
Operator::Lt => gt(stream_values, &last)?,
Operator::LtEq => gt_eq(stream_values, &last)?,
op => return internal_err!("PiecewiseMergeJoin should not contain operator, {op}"),
};
self.join_metrics.probe_hit_rate.add_part(hits.true_count());
Ok(())
}
```
Then `mark_matched_buffered_rows` drops both metric lines and goes back to being purely about the watermark, and the existing order-independence tests still hold (they'd assert `(2, 2)` on row counts rather than batch counts).
If you'd rather not add the comparison, the alternative I'd accept is to leave `probe_hit_rate` unset for existence joins exactly as you've done for `avg_fanout`, and say why in the same comment. `N/A` is honest; `100% (2/2)` for a 3-row-in-16384 hit rate is not.
One related note for whichever way you go: `nothing_left_to_mark()` stops polling the streamed side once the watermark saturates, so the denominator covers only the batches actually read. That's fine and matches `input_rows`, but it's worth a line in the comment so nobody later reads the ratio as covering the whole probe side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "SubhamSinghal (via GitHub)" <gi...@apache.org>.
SubhamSinghal commented on PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#issuecomment-5572900570
@jayzhan211 Thanks for review. Addressed comment in [8758364](https://github.com/apache/datafusion/pull/24956/commits/8758364db05aed1f8061549435fe860767bb6bcb)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "SubhamSinghal (via GitHub)" <gi...@apache.org>.
SubhamSinghal commented on code in PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#discussion_r3943078482
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -336,6 +340,7 @@ impl ClassicPWMJStream {
let buffered_data = Arc::clone(&self.buffered_side.try_as_ready()?.buffered_data);
let buffered_batch = buffered_data.batch();
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
Addressed in https://github.com/apache/datafusion/commit/a0ba9a7897a11f178e009921a5fbb101492ab061
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/existence_join.rs:
##########
@@ -241,12 +241,19 @@ impl ExistencePWMJStream {
self.join_metrics.input_batches.add(1);
self.join_metrics.input_rows.add(batch.num_rows());
- // Only the batch's extreme key is ever compared against the buffered side,
- // so reduce the batch to that one key.
- let stream_values =
- extreme_key(&stream_values, self.sort_option.descending)?;
-
- self.mark_matched_buffered_rows(&stream_values)?;
+ // An empty batch has no extreme key to compare, so it can neither match
+ // nor miss -- counting it either way would understate the real hit rate.
+ if batch.num_rows() > 0 {
+ let join_time = self.join_metrics.join_time.clone();
Review Comment:
Addressed in https://github.com/apache/datafusion/commit/a0ba9a7897a11f178e009921a5fbb101492ab061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#discussion_r3942869583
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/existence_join.rs:
##########
@@ -241,12 +241,19 @@ impl ExistencePWMJStream {
self.join_metrics.input_batches.add(1);
self.join_metrics.input_rows.add(batch.num_rows());
- // Only the batch's extreme key is ever compared against the buffered side,
- // so reduce the batch to that one key.
- let stream_values =
- extreme_key(&stream_values, self.sort_option.descending)?;
-
- self.mark_matched_buffered_rows(&stream_values)?;
+ // An empty batch has no extreme key to compare, so it can neither match
+ // nor miss -- counting it either way would understate the real hit rate.
+ if batch.num_rows() > 0 {
+ let join_time = self.join_metrics.join_time.clone();
+ let join_timer = join_time.timer();
Review Comment:
Do we need either `join_time` or `join_timer` here?
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -336,6 +340,7 @@ impl ClassicPWMJStream {
let buffered_data = Arc::clone(&self.buffered_side.try_as_ready()?.buffered_data);
let buffered_batch = buffered_data.batch();
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
Do we need another timer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#discussion_r3942899931
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -336,6 +340,7 @@ impl ClassicPWMJStream {
let buffered_data = Arc::clone(&self.buffered_side.try_as_ready()?.buffered_data);
let buffered_batch = buffered_data.batch();
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
ditto
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/existence_join.rs:
##########
@@ -241,12 +241,19 @@ impl ExistencePWMJStream {
self.join_metrics.input_batches.add(1);
self.join_metrics.input_rows.add(batch.num_rows());
- // Only the batch's extreme key is ever compared against the buffered side,
- // so reduce the batch to that one key.
- let stream_values =
- extreme_key(&stream_values, self.sort_option.descending)?;
-
- self.mark_matched_buffered_rows(&stream_values)?;
+ // An empty batch has no extreme key to compare, so it can neither match
+ // nor miss -- counting it either way would understate the real hit rate.
+ if batch.num_rows() > 0 {
+ let join_time = self.join_metrics.join_time.clone();
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Pwmj metrics accounting [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on code in PR #24956:
URL: https://github.com/apache/datafusion/pull/24956#discussion_r3942899725
##########
datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs:
##########
@@ -294,6 +295,7 @@ impl ClassicPWMJStream {
}
// Produce more work
+ let join_timer = self.join_metrics.join_time.timer();
Review Comment:
it starts timers inside existing timers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org