You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2024/10/21 04:45:09 UTC
[PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
manuzhang opened a new pull request, #48573:
URL: https://github.com/apache/spark/pull/48573
### What changes were proposed in this pull request?
Add `reportDriverMetrics` method to `Write` API and post custom metrics from driver after v2 write commits.
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
#37205 supported reporting custom driver metrics when reading from v2 table. This is to support writing to v2 table.
### How was this patch tested?
UT.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48573:
URL: https://github.com/apache/spark/pull/48573#discussion_r1842207692
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/Write.java:
##########
@@ -76,4 +77,14 @@ default StreamingWrite toStreaming() {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Returns an array of custom metrics which are collected with values at the driver side only.
+ * Note that these metrics must be included in the supported custom metrics reported by
+ * `supportedCustomMetrics`.
+ */
+ default CustomTaskMetric[] reportDriverMetrics() {
Review Comment:
```suggestion
default CustomTaskMetric [] reportDriverMetrics() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48573:
URL: https://github.com/apache/spark/pull/48573#discussion_r1842208107
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/Write.java:
##########
@@ -76,4 +77,14 @@ default StreamingWrite toStreaming() {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Returns an array of custom metrics which are collected with values at the driver side only.
+ * Note that these metrics must be included in the supported custom metrics reported by
+ * `supportedCustomMetrics`.
+ */
+ default CustomTaskMetric [] reportDriverMetrics() {
Review Comment:
```suggestion
default CustomTaskMetric[] reportDriverMetrics() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48573:
URL: https://github.com/apache/spark/pull/48573#issuecomment-2478369932
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48573:
URL: https://github.com/apache/spark/pull/48573#discussion_r1841450525
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/InMemoryTableMetricSuite.scala:
##########
@@ -67,8 +67,14 @@ class InMemoryTableMetricSuite
statusStore.executionsList().last.metricValues != null)
}
+ val exec = statusStore.executionsList().last
val execId = statusStore.executionsList().last.executionId
Review Comment:
```suggestion
val execId = exec.executionId
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48573:
URL: https://github.com/apache/spark/pull/48573#discussion_r1841452776
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/Write.java:
##########
@@ -76,4 +77,14 @@ default StreamingWrite toStreaming() {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Returns an array of custom metrics which are collected with values at the driver side only.
+ * Note that these metrics must be included in the supported custom metrics reported by
+ * `supportedCustomMetrics`.
+ */
+ default CustomTaskMetric[] reportDriverMetrics() {
Review Comment:
`supportedCustomMetrics` is a noun, how about we name this function `driverSideMetrics`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #48573:
URL: https://github.com/apache/spark/pull/48573#issuecomment-2477799592
@manuzhang can you rebase with the latest master and re-trigger the CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #48573:
URL: https://github.com/apache/spark/pull/48573#issuecomment-2478071659
@cloud-fan Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #48573: [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table
URL: https://github.com/apache/spark/pull/48573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48573:
URL: https://github.com/apache/spark/pull/48573#discussion_r1841454152
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/Write.java:
##########
@@ -76,4 +77,14 @@ default StreamingWrite toStreaming() {
default CustomMetric[] supportedCustomMetrics() {
return new CustomMetric[]{};
}
+
+ /**
+ * Returns an array of custom metrics which are collected with values at the driver side only.
+ * Note that these metrics must be included in the supported custom metrics reported by
+ * `supportedCustomMetrics`.
+ */
+ default CustomTaskMetric[] reportDriverMetrics() {
Review Comment:
oh nvm, this is to be consistent with the read side API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #48573:
URL: https://github.com/apache/spark/pull/48573#issuecomment-2434446787
gentle ping @viirya @cloud-fan for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50049][SQL] Support custom driver metrics in writing to v2 table [spark]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #48573:
URL: https://github.com/apache/spark/pull/48573#issuecomment-2425575029
cc @karuppayya @viirya @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org