You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "dcoliversun (via GitHub)" <gi...@apache.org> on 2024/09/04 03:40:46 UTC
[PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
dcoliversun opened a new pull request, #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111
## What changes were proposed in this pull request?
This PR aims to support IncrementMetric in gluten. In delta, IncrementMetric is used:
* DeleteCommand: Filter
* https://github.com/delta-io/delta/blob/cdd39dda25f1a9b06ce7a414d0ebff4ddfa419c5/spark/src/main/scala/org/apache/spark/sql/delta/commands/DeleteCommand.scala#L311
* https://github.com/delta-io/delta/blob/cdd39dda25f1a9b06ce7a414d0ebff4ddfa419c5/spark/src/main/scala/org/apache/spark/sql/delta/commands/DeleteCommand.scala#L461
* UpdateCommand: Filter
* https://github.com/delta-io/delta/blob/cdd39dda25f1a9b06ce7a414d0ebff4ddfa419c5/spark/src/main/scala/org/apache/spark/sql/delta/commands/UpdateCommand.scala#L188
* https://github.com/delta-io/delta/blob/cdd39dda25f1a9b06ce7a414d0ebff4ddfa419c5/spark/src/main/scala/org/apache/spark/sql/delta/commands/UpdateCommand.scala#L366
(Fixes: \#7110)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1759650448
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
Thank you for your help. The ClickHouse unit tests are OK. I am currently conducting end-to-end testing and investigating the reason for a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2339541627
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2360265169
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2342681666
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2331013325
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2406590748
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2407309557
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2360362889
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2345425166
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1753391340
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
@yma11 Updated. And some Clickhouse unit tests failed in Jenkins. Could you help me to check exception information? I have no account.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2723839621
@felipepessoto Got it. Looks like that you use new test? I will see it soon. Could you please submit an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2327851596
https://github.com/apache/incubator-gluten/issues/7110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2337510050
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2337627975
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2410640736
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2339563812
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2345141798
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2723466481
@dcoliversun, I think I found a bug with this change and FilterMetricsUpdater. The DeleteCommand query is:
data.filter(new Column(cond))
.select(input_file_name())
.filter(new Column(incrDeletedCountExpr))
.distinct()
.as[String]
.collect()
If you remove the `.select(input_file_name())`, and replace it by something like `.select(col("_metadata.file_path"))`, or `.select(org.apache.spark.sql.functions.lit("TEST"))` the FilterMetricsUpdater.updateNativeMetrics is not called with correct `operatorMetrics.outputRows`.
I guess some shortcut is taken when we don't have a UDF. But this shouldn't happen, since IncrementMetric is Nondeterministic. I'm not sure if this is the case though, Idk how to troubleshoot this.
I added some println:
```
class FilterMetricsUpdater(
val metrics: Map[String, SQLMetric],
val extraMetrics: Seq[(String, SQLMetric)])
extends MetricsUpdater {
override def updateNativeMetrics(opMetrics: IOperatorMetrics): Unit = {
println("FilterMetricsUpdater: 1")
if (opMetrics != null) {
println("FilterMetricsUpdater: 2")
val operatorMetrics = opMetrics.asInstanceOf[OperatorMetrics]
metrics("numOutputRows") += operatorMetrics.outputRows
metrics("outputVectors") += operatorMetrics.outputVectors
metrics("outputBytes") += operatorMetrics.outputBytes
metrics("cpuCount") += operatorMetrics.cpuCount
metrics("wallNanos") += operatorMetrics.wallNanos
metrics("peakMemoryBytes") += operatorMetrics.peakMemoryBytes
metrics("numMemoryAllocations") += operatorMetrics.numMemoryAllocations
println(s"FilterMetricsUpdater: 3. ${extraMetrics.size}. $extraMetrics")
extraMetrics.foreach {
case (name, metric) =>
name match {
case "increment_metric" =>
println(s"FilterMetricsUpdater: 4. ${operatorMetrics.outputRows}.")
metric += operatorMetrics.outputRows
case _ => // do nothing
}
}
}
}
}
```
This is what it shows for `.select(org.apache.spark.sql.functions.lit("TEST"))` or `.select(col("_metadata.file_path"))` and run `build/sbt 'spark/testOnly org.apache.spark.sql.delta.DescribeDeltaHistorySuite -- -t "operation metrics - delete"'`
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 0.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 0.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
This is when we use the input_file_name:
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 0.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 1.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 0.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2134, name: Some(number of rows deleted.), value: 0)))
FilterMetricsUpdater: 4. 0.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 1. List((increment_metric,SQLMetric(id: 2150, name: Some(number of rows touched), value: 0)))
FilterMetricsUpdater: 4. 3.
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
FilterMetricsUpdater: 1
FilterMetricsUpdater: 2
FilterMetricsUpdater: 3. 0. List()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2342671659
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1751417842
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
Whether can we put more code in `FilterExecTransformerBase` if delta can share the logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1796887527
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
Will be fixed in #7483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2725637985
Thanks, I opened it here: https://github.com/apache/incubator-gluten/issues/9003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2327851781
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1743137633
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicPhysicalOperatorTransformer.scala:
##########
@@ -73,9 +77,17 @@ abstract class FilterExecTransformerBase(val cond: Expression, val input: SparkP
validation: Boolean): RelNode = {
assert(condExpr != null)
val args = context.registeredFunction
- val condExprNode = ExpressionConverter
- .replaceWithExpressionTransformer(condExpr, attributeSeq = originalInputAttributes)
- .doTransform(args)
+ val condExprNode = condExpr match {
+ case IncrementMetric(child, metric) =>
Review Comment:
Can we create a class similar as `CHFilterExecTransformer`, such `DeltaFilterExecTransformer` and override `getRelNode` method instead? We should put it under `gluten-delta` folder thus to keep core clean from data lake specific logic and no need add delta package as dependency for `gluten-substrait`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 merged PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1755947686
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
[log.txt](https://github.com/user-attachments/files/16972027/log.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "yma11 (via GitHub)" <gi...@apache.org>.
yma11 commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1797511833
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicPhysicalOperatorTransformer.scala:
##########
@@ -160,7 +178,7 @@ object FilterExecTransformerBase {
}
}
-case class ProjectExecTransformer private (projectList: Seq[NamedExpression], child: SparkPlan)
+abstract class ProjectExecTransformerBase(val list: Seq[NamedExpression], val input: SparkPlan)
Review Comment:
@zhztheplayer here is a change from case class to abstract class, any concerns about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2345414465
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1751705705
##########
gluten-delta/src/main/delta-32/org/apache/gluten/execution/DeltaFilterExecTransformer.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.execution
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.expression.{ConverterUtils, ExpressionConverter}
+import org.apache.gluten.metrics.MetricsUpdater
+import org.apache.gluten.substrait.`type`.TypeBuilder
+import org.apache.gluten.substrait.SubstraitContext
+import org.apache.gluten.substrait.extensions.ExtensionBuilder
+import org.apache.gluten.substrait.rel.{RelBuilder, RelNode}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Expression}
+import org.apache.spark.sql.delta.metric.IncrementMetric
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.metric.SQLMetric
+
+import scala.collection.JavaConverters._
+
+case class DeltaFilterExecTransformer(condition: Expression, child: SparkPlan)
Review Comment:
Sure, I am doing on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2342679958
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#discussion_r1743190286
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BasicPhysicalOperatorTransformer.scala:
##########
@@ -73,9 +77,17 @@ abstract class FilterExecTransformerBase(val cond: Expression, val input: SparkP
validation: Boolean): RelNode = {
assert(condExpr != null)
val args = context.registeredFunction
- val condExprNode = ExpressionConverter
- .replaceWithExpressionTransformer(condExpr, attributeSeq = originalInputAttributes)
- .doTransform(args)
+ val condExprNode = condExpr match {
+ case IncrementMetric(child, metric) =>
Review Comment:
Sure. I will do it after Project supports IncrementMetric
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2330544038
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2327976011
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2330812863
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2345766464
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-7110][VL][DELTA] support IncrementMetric in gluten [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7111:
URL: https://github.com/apache/incubator-gluten/pull/7111#issuecomment-2360290808
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org