You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "avantgardnerio (via GitHub)" <gi...@apache.org> on 2023/01/26 19:56:29 UTC
[GitHub] [arrow-ballista] avantgardnerio opened a new pull request, #634: Upgrade to datafusion 16
avantgardnerio opened a new pull request, #634:
URL: https://github.com/apache/arrow-ballista/pull/634
# Which issue does this PR close?
Closes #610.
# Rationale for this change
Described in issue.
# What changes are included in this PR?
DataFusion 16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] andygrove commented on a diff in pull request #634: Upgrade to datafusion 16
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #634:
URL: https://github.com/apache/arrow-ballista/pull/634#discussion_r1088420670
##########
ballista/scheduler/src/state/execution_graph_dot.rs:
##########
@@ -548,87 +549,75 @@ filter_expr="]
stage_2_0_0 -> stage_2_0
}
subgraph cluster2 {
- label = "Stage 3 [Resolved]";
- stage_3_0 [shape=box, label="ShuffleWriter [0 partitions]"]
- stage_3_0_0 [shape=box, label="MemoryExec"]
+ label = "Stage 3 [Unresolved]";
+ stage_3_0 [shape=box, label="ShuffleWriter [48 partitions]"]
+ stage_3_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0 [shape=box, label="HashJoin
+join_expr=a@0 = a@0
+filter_expr="]
+ stage_3_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=1]"]
+ stage_3_0_0_0_0_0 -> stage_3_0_0_0_0
+ stage_3_0_0_0_0 -> stage_3_0_0_0
+ stage_3_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=2]"]
+ stage_3_0_0_0_1_0 -> stage_3_0_0_0_1
+ stage_3_0_0_0_1 -> stage_3_0_0_0
+ stage_3_0_0_0 -> stage_3_0_0
stage_3_0_0 -> stage_3_0
}
subgraph cluster3 {
- label = "Stage 4 [Unresolved]";
- stage_4_0 [shape=box, label="ShuffleWriter [48 partitions]"]
- stage_4_0_0 [shape=box, label="Projection: a@0, a@1, a@2"]
- stage_4_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0 [shape=box, label="HashJoin
+ label = "Stage 4 [Resolved]";
+ stage_4_0 [shape=box, label="ShuffleWriter [0 partitions]"]
+ stage_4_0_0 [shape=box, label="MemoryExec"]
+ stage_4_0_0 -> stage_4_0
+ }
+ subgraph cluster4 {
+ label = "Stage 5 [Unresolved]";
+ stage_5_0 [shape=box, label="ShuffleWriter [48 partitions]"]
+ stage_5_0_0 [shape=box, label="Projection: a@0, a@1, a@2"]
+ stage_5_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0 [shape=box, label="HashJoin
join_expr=a@1 = a@0
filter_expr="]
- stage_4_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0 [shape=box, label="HashJoin
-join_expr=a@0 = a@0
-filter_expr="]
- stage_4_0_0_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=1]"]
- stage_4_0_0_0_0_0_0_0_0 -> stage_4_0_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0_0 -> stage_4_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=2]"]
- stage_4_0_0_0_0_0_0_1_0 -> stage_4_0_0_0_0_0_0_1
- stage_4_0_0_0_0_0_0_1 -> stage_4_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0 -> stage_4_0_0_0_0_0
- stage_4_0_0_0_0_0 -> stage_4_0_0_0_0
- stage_4_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=3]"]
- stage_4_0_0_0_0_1_0 -> stage_4_0_0_0_0_1
- stage_4_0_0_0_0_1 -> stage_4_0_0_0_0
- stage_4_0_0_0_0 -> stage_4_0_0_0
- stage_4_0_0_0 -> stage_4_0_0
- stage_4_0_0 -> stage_4_0
+ stage_5_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=3]"]
+ stage_5_0_0_0_0_0_0 -> stage_5_0_0_0_0_0
+ stage_5_0_0_0_0_0 -> stage_5_0_0_0_0
+ stage_5_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=4]"]
+ stage_5_0_0_0_0_1_0 -> stage_5_0_0_0_0_1
+ stage_5_0_0_0_0_1 -> stage_5_0_0_0_0
+ stage_5_0_0_0_0 -> stage_5_0_0_0
+ stage_5_0_0_0 -> stage_5_0_0
+ stage_5_0_0 -> stage_5_0
}
- stage_1_0 -> stage_4_0_0_0_0_0_0_0_0
- stage_2_0 -> stage_4_0_0_0_0_0_0_1_0
- stage_3_0 -> stage_4_0_0_0_0_1_0
+ stage_1_0 -> stage_3_0_0_0_0_0
+ stage_2_0 -> stage_3_0_0_0_1_0
+ stage_3_0 -> stage_5_0_0_0_0_0_0
+ stage_4_0 -> stage_5_0_0_0_0_1_0
}
-"#;
- assert_eq!(expected, &dot);
+"#
+ .trim();
+ assert_eq!(dot.trim(), expected);
Ok(())
}
#[tokio::test]
async fn query_stage_optimized() -> Result<()> {
let graph = test_graph_optimized().await?;
let dot = ExecutionGraphDot::generate_for_query_stage(Arc::new(graph), 4)
Review Comment:
The plan has changed a lot due to optimizations DF ... I think looking at stage 3 or 5 here rather than 4 would be more interesting though (those have joins)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] avantgardnerio commented on a diff in pull request #634: Upgrade to datafusion 16
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on code in PR #634:
URL: https://github.com/apache/arrow-ballista/pull/634#discussion_r1088425933
##########
ballista/scheduler/src/state/execution_graph_dot.rs:
##########
@@ -548,87 +549,75 @@ filter_expr="]
stage_2_0_0 -> stage_2_0
}
subgraph cluster2 {
- label = "Stage 3 [Resolved]";
- stage_3_0 [shape=box, label="ShuffleWriter [0 partitions]"]
- stage_3_0_0 [shape=box, label="MemoryExec"]
+ label = "Stage 3 [Unresolved]";
+ stage_3_0 [shape=box, label="ShuffleWriter [48 partitions]"]
+ stage_3_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0 [shape=box, label="HashJoin
+join_expr=a@0 = a@0
+filter_expr="]
+ stage_3_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=1]"]
+ stage_3_0_0_0_0_0 -> stage_3_0_0_0_0
+ stage_3_0_0_0_0 -> stage_3_0_0_0
+ stage_3_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_3_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=2]"]
+ stage_3_0_0_0_1_0 -> stage_3_0_0_0_1
+ stage_3_0_0_0_1 -> stage_3_0_0_0
+ stage_3_0_0_0 -> stage_3_0_0
stage_3_0_0 -> stage_3_0
}
subgraph cluster3 {
- label = "Stage 4 [Unresolved]";
- stage_4_0 [shape=box, label="ShuffleWriter [48 partitions]"]
- stage_4_0_0 [shape=box, label="Projection: a@0, a@1, a@2"]
- stage_4_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0 [shape=box, label="HashJoin
+ label = "Stage 4 [Resolved]";
+ stage_4_0 [shape=box, label="ShuffleWriter [0 partitions]"]
+ stage_4_0_0 [shape=box, label="MemoryExec"]
+ stage_4_0_0 -> stage_4_0
+ }
+ subgraph cluster4 {
+ label = "Stage 5 [Unresolved]";
+ stage_5_0 [shape=box, label="ShuffleWriter [48 partitions]"]
+ stage_5_0_0 [shape=box, label="Projection: a@0, a@1, a@2"]
+ stage_5_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0 [shape=box, label="HashJoin
join_expr=a@1 = a@0
filter_expr="]
- stage_4_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0 [shape=box, label="HashJoin
-join_expr=a@0 = a@0
-filter_expr="]
- stage_4_0_0_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=1]"]
- stage_4_0_0_0_0_0_0_0_0 -> stage_4_0_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0_0 -> stage_4_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=2]"]
- stage_4_0_0_0_0_0_0_1_0 -> stage_4_0_0_0_0_0_0_1
- stage_4_0_0_0_0_0_0_1 -> stage_4_0_0_0_0_0_0
- stage_4_0_0_0_0_0_0 -> stage_4_0_0_0_0_0
- stage_4_0_0_0_0_0 -> stage_4_0_0_0_0
- stage_4_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=4096]"]
- stage_4_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=3]"]
- stage_4_0_0_0_0_1_0 -> stage_4_0_0_0_0_1
- stage_4_0_0_0_0_1 -> stage_4_0_0_0_0
- stage_4_0_0_0_0 -> stage_4_0_0_0
- stage_4_0_0_0 -> stage_4_0_0
- stage_4_0_0 -> stage_4_0
+ stage_5_0_0_0_0_0 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0_0_0 [shape=box, label="UnresolvedShuffleExec [stage_id=3]"]
+ stage_5_0_0_0_0_0_0 -> stage_5_0_0_0_0_0
+ stage_5_0_0_0_0_0 -> stage_5_0_0_0_0
+ stage_5_0_0_0_0_1 [shape=box, label="CoalesceBatches [batchSize=8192]"]
+ stage_5_0_0_0_0_1_0 [shape=box, label="UnresolvedShuffleExec [stage_id=4]"]
+ stage_5_0_0_0_0_1_0 -> stage_5_0_0_0_0_1
+ stage_5_0_0_0_0_1 -> stage_5_0_0_0_0
+ stage_5_0_0_0_0 -> stage_5_0_0_0
+ stage_5_0_0_0 -> stage_5_0_0
+ stage_5_0_0 -> stage_5_0
}
- stage_1_0 -> stage_4_0_0_0_0_0_0_0_0
- stage_2_0 -> stage_4_0_0_0_0_0_0_1_0
- stage_3_0 -> stage_4_0_0_0_0_1_0
+ stage_1_0 -> stage_3_0_0_0_0_0
+ stage_2_0 -> stage_3_0_0_0_1_0
+ stage_3_0 -> stage_5_0_0_0_0_0_0
+ stage_4_0 -> stage_5_0_0_0_0_1_0
}
-"#;
- assert_eq!(expected, &dot);
+"#
+ .trim();
+ assert_eq!(dot.trim(), expected);
Ok(())
}
#[tokio::test]
async fn query_stage_optimized() -> Result<()> {
let graph = test_graph_optimized().await?;
let dot = ExecutionGraphDot::generate_for_query_stage(Arc::new(graph), 4)
Review Comment:
I switched it to 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] Dandandan commented on pull request #634: Upgrade to datafusion 16
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #634:
URL: https://github.com/apache/arrow-ballista/pull/634#issuecomment-1408176649
Thanks [avantgardnerio](https://github.com/avantgardnerio) #612 got fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] Dandandan closed pull request #634: Upgrade to datafusion 16
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed pull request #634: Upgrade to datafusion 16
URL: https://github.com/apache/arrow-ballista/pull/634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org