You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/08/28 20:19:48 UTC
[PR] fix: shuffle reader should return statistics [datafusion-ballista]
milenkovicm opened a new pull request, #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302
# Which issue does this PR close?
Closes #.
# Rationale for this change
with datafusion 48 method which is to be used for Exec statistic information changed. ShuffleReader did not update to reflect change hence no statistic has been propagated to query rules, probably yielding suboptimal plans.
# What changes are included in this PR?
- use `partition_statistics` for to provide `ShuffleReader` statistics
- provide tests for them
- method `UnresolvedStage::to_resolved` to user provided configuration option rather than hard coded one making optimiser rules used in it configurable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "ZihuanLing (via GitHub)" <gi...@apache.org>.
ZihuanLing commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239377541
Thanks you for openning the issue 🌹
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239390521
https://github.com/apache/datafusion-ballista/commit/bca73ad2ee298b2a97e57366b894a5425fb71947 provides custom rule for ballista fixing this ussue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm merged PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3240364204
@metegenez would you be able to review this change when you're around (and few others open)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "ZihuanLing (via GitHub)" <gi...@apache.org>.
ZihuanLing commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239033243
@milenkovicm hi, I've applied this patch to my code, and tried to run tpcds query 44 and 54, I meet this error :
Error: ArrowError(ExternalError(Execution("Job mAxRhlg failed: Job failed due to stage 18 failed: Task failed due to runtime execution error: DataFusionError(Internal(\"Invalid NestedLoopJoinExec, the output partition count of the left child must be 1,consider using CoalescePartitionsExec or the EnforceDistribution rule\"))\n")), None)
here's some logs maybe helpful: (just part of logs)
```
=========UnResolvedStage[stage_id=18.0, children=2]=========
Inputs{16: StageOutput { partition_locations: {}, complete: false }, 17: StageOutput { partition_locations: {}, complete: false }}
ShuffleWriterExec: partitions:None
NestedLoopJoinExec: join_type=Inner, filter=CAST(d_month_seq@0 AS Int64) >= date_dim.d_month_seq + Int64(1)@1, projection=[c_customer_sk@0, ss_ext_sales_price@1, d_month_seq@2]
CoalescePartitionsExec
UnresolvedShuffleExec: partitions=Hash([UnKnownColumn { name: "d_date_sk@3" }], 4)
AggregateExec: mode=FinalPartitioned, gby=[date_dim.d_month_seq + Int64(1)@0 as date_dim.d_month_seq + Int64(1)], aggr=[]
CoalesceBatchesExec: target_batch_size=4096
UnresolvedShuffleExec: partitions=Hash([Column { name: "date_dim.d_month_seq + Int64(1)", index: 0 }], 4)
...
=========UnResolvedStage[stage_id=18.0, children=2]=========
Inputs{17: StageOutput { partition_locations: {}, complete: false }, 16: StageOutput { partition_locations: {}, complete: false }}
ShuffleWriterExec: partitions:None
NestedLoopJoinExec: join_type=Inner, filter=CAST(d_month_seq@0 AS Int64) >= date_dim.d_month_seq + Int64(1)@1, projection=[c_customer_sk@0, ss_ext_sales_price@1, d_month_seq@2]
CoalescePartitionsExec
UnresolvedShuffleExec: partitions=Hash([UnKnownColumn { name: "d_date_sk@3" }], 4)
AggregateExec: mode=FinalPartitioned, gby=[date_dim.d_month_seq + Int64(1)@0 as date_dim.d_month_seq + Int64(1)], aggr=[]
CoalesceBatchesExec: target_batch_size=4096
UnresolvedShuffleExec: partitions=Hash([Column { name: "date_dim.d_month_seq + Int64(1)", index: 0 }], 4)
...
=========ResolvedStage[stage_id=18.0, partitions=1]=========
ShuffleWriterExec: partitions:Some(Hash([Column { name: "p_promo_sk", index: 0 }], 4))
DataSourceExec: file_groups={1 group: [[opt/lzh/10G/promotion.dat]]}, projection=[p_promo_sk], file_type=csv, has_header=false
...
=========UnResolvedStage[stage_id=18.0, children=1]=========
Inputs{17: StageOutput { partition_locations: {}, complete: false }}
ShuffleWriterExec: partitions:None
SortExec: expr=[return_ratio@1 ASC NULLS LAST], preserve_partitioning=[true]
ProjectionExec: expr=[ss_item_sk@0 as item, CAST(sum(coalesce(sr.sr_return_quantity,Int64(0)))@1 AS Decimal128(15, 4)) / CAST(sum(coalesce(sts.ss_quantity,Int64(0)))@2 AS Decimal128(15, 4)) as return_ratio, CAST(sum(coalesce(sr.sr_return_amt,Int64(0)))@3 AS Decimal128(15, 4)) / CAST(sum(coalesce(sts.ss_net_paid,Int64(0)))@4 AS Decimal128(15, 4)) as currency_ratio]
AggregateExec: mode=FinalPartitioned, gby=[ss_item_sk@0 as ss_item_sk], aggr=[sum(coalesce(sr.sr_return_quantity,Int64(0))), sum(coalesce(sts.ss_quantity,Int64(0))), sum(coalesce(sr.sr_return_amt,Int64(0))), sum(coalesce(sts.ss_net_paid,Int64(0)))]
CoalesceBatchesExec: target_batch_size=4096
UnresolvedShuffleExec: partitions=Hash([Column { name: "ss_item_sk", index: 0 }], 4)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239281415
hey @ZihuanLing thanks for reporting
Looks like that enabling statistics will trigger this issue. I believe this is a bug in datafusion code, as the swap method or create NLJ does not check right partitioning before swapping. Will open bug on datafusion and see from there.
Important this is not issue created by this patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3239308351
I have opened a bug in datafusion https://github.com/apache/datafusion/issues/17358
If it is bug with datafusion ballista does not need to change (I'll push temporary rule before ballista 49 release to fix this issue until datafusion 50 update)
if it is not bug with datafusion, ballista will have to create its own `JoinSelection`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `ShuffleReader` should return statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1302:
URL: https://github.com/apache/datafusion-ballista/pull/1302#issuecomment-3240211869
after investigation this is not issue I believe this is not an issue with datafusion, its ballista issue. There is a #1304 explaining details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org