You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/27 19:44:57 UTC
[GitHub] [spark] EnricoMi opened a new pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
EnricoMi opened a new pull request #33545:
URL: https://github.com/apache/spark/pull/33545
### What changes were proposed in this pull request?
As @HyukjinKwon pointed out, the Observation API (Scala, Java, PySpark) could return a `Map` / `Dict`. It currently returns `Row` simply because the metrics are (internal to Observation) retrieved from the listener as rows. Since that is hidden from the user by the Observation API, there is no need to return `Row`.
While touching this code, this moves the unit tests from `DataFrameSuite,scala` to `DatasetSuite.scala` and from `JavaDataFrameSuite.java` to `JavaDatasetSuite.java`, which is a better place.
### Why are the changes needed?
This simplifies the API and accessing the metrics, especially in Java. There is no need for the concept `Row` when retrieving the observation result.
### Does this PR introduce _any_ user-facing change?
Yes, it replaces `get` with `getAsRow` and introduces `getAsMap`, `getAsJavaMap`, `getAsDict`. So this removes `Observation.get`, but this has just been added in the current 3.3.0 version.
### How was this patch tested?
This is tested in `DatasetSuite.SPARK-34806: observation on datasets`, `JavaDatasetSuite.testObservation` and `test_dataframe.test_observe`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889815158
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888340530
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679561495
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -75,23 +79,39 @@ class Observation(name: String) {
}
/**
- * Get the observed metrics. This waits for the observed dataset to finish its first action.
- * Only the result of the first action is available. Subsequent actions do not modify the result.
+ * (Scala-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
+ *
+ * @return the observed metrics as a `java.util.Map[String, Object]`
+ * @throws InterruptedException interrupted while waiting
+ */
+ @throws[InterruptedException]
+ def getAsJava: java.util.Map[String, Object] = {
Review comment:
Here too. BTW, I remember `AnyRef` corresponds to `Object`. If `Map[String, _]` doesn't work, can we switch to `AnyRef`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888368298
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679137829
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
the observed metrics
"""
assert self._jo is not None, 'call DataFrame.observe'
- jrow = self._jo.get()
- return self._to_row(jrow)
+ jmap = self._jo.getAsJavaMap()
+ # return a pure Python dict, not a py4j JavaMap
+ return {k: v for k, v in jmap.items()}
Review comment:
no, that is a JavaMap, a py4j implementation of dict that syncs changes of the Python dict back to the Java map inside the jvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889934018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141892/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889147106
**[Test build #141844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141844/testReport)** for PR 33545 at commit [`0b863e0`](https://github.com/apache/spark/commit/0b863e032d723162502f01e011c9db258c519b4a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889922030
**[Test build #141892 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141892/testReport)** for PR 33545 at commit [`80e35d9`](https://github.com/apache/spark/commit/80e35d94c545dc1f03a70107467c3e92b3d19e29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889242927
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888525468
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889769745
**[Test build #141892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141892/testReport)** for PR 33545 at commit [`80e35d9`](https://github.com/apache/spark/commit/80e35d94c545dc1f03a70107467c3e92b3d19e29).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r680647416
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -701,6 +701,57 @@ class DatasetSuite extends QueryTest
1 -> "a", 2 -> "bc", 3 -> "d")
}
+ test("SPARK-34806: observation on datasets") {
+ val namedObservation = Observation("named")
+ val unnamedObservation = Observation()
+
+ val df = spark.range(100)
+ val observed_df = df
+ .observe(
+ namedObservation,
+ min($"id").as("min_val"),
+ max($"id").as("max_val"),
+ sum($"id").as("sum_val"),
Review comment:
one minor comment: what happens if there are duplicated names like `min(...).as("a"), max(...).as("a")`? Do we silently drop one value or do we fail at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888156614
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887965356
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889194105
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141844/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888082838
@HyukjinKwon `get` now returns the Scala Map, `getAsRow` is removed. I would still like to keep `getAsJavaMap` as it simplifies usage in Java (see JavaDatasetSuite.java`.
There is one issue with returning a `Map` though: The map will lose some metrics when the aggregation expressions have duplicate column names:
> df.observe(observation, lit(1).as("a"), lit(2).as("a")).count
> observation.get
Map(a -> 2)
Should I add some prefix to the duplicate column names?
Map(a_1 -> 1, a_2 -> 2)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887809836
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889192411
**[Test build #141844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141844/testReport)** for PR 33545 at commit [`0b863e0`](https://github.com/apache/spark/commit/0b863e032d723162502f01e011c9db258c519b4a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679739875
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -75,23 +79,39 @@ class Observation(name: String) {
}
/**
- * Get the observed metrics. This waits for the observed dataset to finish its first action.
- * Only the result of the first action is available. Subsequent actions do not modify the result.
+ * (Scala-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888237797
**[Test build #141759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141759/testReport)** for PR 33545 at commit [`9f81b32`](https://github.com/apache/spark/commit/9f81b3219beb2c71d3f50e5be33520012b681bc6).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889831897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889194105
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141844/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r678091095
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -82,7 +86,7 @@ class Observation(name: String) {
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def getAsRow: Row = {
Review comment:
done, but the getAsJavaMap is still useful in two ways
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888309535
**[Test build #141776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141776/testReport)** for PR 33545 at commit [`32c6ad5`](https://github.com/apache/spark/commit/32c6ad5fcdc038620fc9d5d15036115485e61f7a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888156614
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888134884
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887918956
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888081989
**[Test build #141759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141759/testReport)** for PR 33545 at commit [`9f81b32`](https://github.com/apache/spark/commit/9f81b3219beb2c71d3f50e5be33520012b681bc6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888239012
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889769745
**[Test build #141892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141892/testReport)** for PR 33545 at commit [`80e35d9`](https://github.com/apache/spark/commit/80e35d94c545dc1f03a70107467c3e92b3d19e29).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888239012
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141759/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887875016
OK to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887885554
**[Test build #141736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141736/testReport)** for PR 33545 at commit [`8d7aa97`](https://github.com/apache/spark/commit/8d7aa978269cb4f1a271dca3040bec6b6becbcb0).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887874923
Yeah I think we don't necessarily have to use Row here. Cc @hvanhovell too fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889934018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141892/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889291162
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-890430408
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888081989
**[Test build #141759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141759/testReport)** for PR 33545 at commit [`9f81b32`](https://github.com/apache/spark/commit/9f81b3219beb2c71d3f50e5be33520012b681bc6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679132200
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
Review comment:
you are right, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888309535
**[Test build #141776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141776/testReport)** for PR 33545 at commit [`32c6ad5`](https://github.com/apache/spark/commit/32c6ad5fcdc038620fc9d5d15036115485e61f7a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679141050
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific)
+ * Get the observed metrics. This waits for the observed dataset to finish its first action.
+ * Only the result of the first action is available. Subsequent actions do not modify the result.
+ *
Review comment:
applied
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
Review comment:
added
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific)
+ * Get the observed metrics. This waits for the observed dataset to finish its first action.
+ * Only the result of the first action is available. Subsequent actions do not modify the result.
+ *
+ * @return the observed metrics as a `java.util.Map[String, Object]`
+ * @throws InterruptedException interrupted while waiting
+ */
+ @throws[InterruptedException]
+ def getAsJavaMap: java.util.Map[String, Object] = {
Review comment:
changed
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
the observed metrics
"""
assert self._jo is not None, 'call DataFrame.observe'
- jrow = self._jo.get()
- return self._to_row(jrow)
+ jmap = self._jo.getAsJavaMap()
Review comment:
done
##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
##########
@@ -391,6 +390,60 @@ public void testGroupBy() {
Assert.assertEquals(asSet("1a#2", "3foobar#6", "5#10"), toSet(cogrouped.collectAsList()));
}
+ @Test
+ public void testObservation() {
+ // SPARK-34806: tests the Observation Java API and Dataset.observe(Observation, Column, Column*)
+ Observation namedObservation = new Observation("named");
+ Observation unnamedObservation = new Observation();
+
+ Dataset<Long> df = spark
+ .range(100)
+ .observe(
+ namedObservation,
+ min(col("id")).as("min_val"),
+ max(col("id")).as("max_val"),
+ sum(col("id")).as("sum_val"),
+ count(when(pmod(col("id"), lit(2)).$eq$eq$eq(0), 1)).as("num_even")
+ )
+ .observe(
+ unnamedObservation,
+ avg(col("id")).cast("int").as("avg_val")
+ );
+
+ df.collect();
+ Map<String, Object> namedMetrics = null;
+ Map<String, Object> unnamedMetrics = null;
+
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679142250
##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
##########
@@ -391,6 +390,60 @@ public void testGroupBy() {
Assert.assertEquals(asSet("1a#2", "3foobar#6", "5#10"), toSet(cogrouped.collectAsList()));
}
+ @Test
+ public void testObservation() {
+ // SPARK-34806: tests the Observation Java API and Dataset.observe(Observation, Column, Column*)
+ Observation namedObservation = new Observation("named");
+ Observation unnamedObservation = new Observation();
+
+ Dataset<Long> df = spark
+ .range(100)
+ .observe(
+ namedObservation,
+ min(col("id")).as("min_val"),
+ max(col("id")).as("max_val"),
+ sum(col("id")).as("sum_val"),
+ count(when(pmod(col("id"), lit(2)).$eq$eq$eq(0), 1)).as("num_even")
+ )
+ .observe(
+ unnamedObservation,
+ avg(col("id")).cast("int").as("avg_val")
+ );
+
+ df.collect();
+ Map<String, Object> namedMetrics = null;
+ Map<String, Object> unnamedMetrics = null;
+
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
+ } catch (InterruptedException e) {
+ Assert.fail();
+ }
+ Map<String, Object> expectedNamedMetrics = new HashMap<String, Object>() {{
+ put("min_val", 0L);
+ put("max_val", 99L);
+ put("sum_val", 4950L);
+ put("num_even", 50L);
+ }};
+ Assert.assertEquals(expectedNamedMetrics, namedMetrics);
+
+ Map<String, Object> expectedUnnamedMetrics = new HashMap<String, Object>() {{
+ put("avg_val", 49);
+ }};
+ Assert.assertEquals(expectedUnnamedMetrics, unnamedMetrics);
+
+ // we can get the result multiple times
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888772199
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887786544
@HyukjinKwon @cloud-fan if there is no value in `getAsRow`, then this could be removed entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888368264
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888368298
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46288/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889797173
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-890479742
@HyukjinKwon thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889291162
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889285475
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46359/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888500466
**[Test build #141776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141776/testReport)** for PR 33545 at commit [`32c6ad5`](https://github.com/apache/spark/commit/32c6ad5fcdc038620fc9d5d15036115485e61f7a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888525468
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141776/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679560539
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -75,23 +79,39 @@ class Observation(name: String) {
}
/**
- * Get the observed metrics. This waits for the observed dataset to finish its first action.
- * Only the result of the first action is available. Subsequent actions do not modify the result.
+ * (Scala-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
Review comment:
sorry last comment. can we use sth like `Map[String, _]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888772199
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887902783
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-888111381
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46270/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r680673878
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -701,6 +701,57 @@ class DatasetSuite extends QueryTest
1 -> "a", 2 -> "bc", 3 -> "d")
}
+ test("SPARK-34806: observation on datasets") {
+ val namedObservation = Observation("named")
+ val unnamedObservation = Observation()
+
+ val df = spark.range(100)
+ val observed_df = df
+ .observe(
+ namedObservation,
+ min($"id").as("min_val"),
+ max($"id").as("max_val"),
+ sum($"id").as("sum_val"),
Review comment:
Yes, It behaves identical to `Row.getValuesMap(Row.schema.fieldNames)`, which drops all but the last occurrence of a column name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887876643
**[Test build #141736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141736/testReport)** for PR 33545 at commit [`8d7aa97`](https://github.com/apache/spark/commit/8d7aa978269cb4f1a271dca3040bec6b6becbcb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887918956
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679145596
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
the observed metrics
"""
assert self._jo is not None, 'call DataFrame.observe'
- jrow = self._jo.get()
- return self._to_row(jrow)
+ jmap = self._jo.getAsJavaMap()
+ # return a pure Python dict, not a py4j JavaMap
+ return {k: v for k, v in jmap.items()}
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887965356
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887809836
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r677842548
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -82,7 +86,7 @@ class Observation(name: String) {
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def getAsRow: Row = {
Review comment:
Can we just change the return type alone instead of adding other new methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887897856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33545:
URL: https://github.com/apache/spark/pull/33545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887897856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141736/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889147106
**[Test build #141844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141844/testReport)** for PR 33545 at commit [`0b863e0`](https://github.com/apache/spark/commit/0b863e032d723162502f01e011c9db258c519b4a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887917209
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46249/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] EnricoMi commented on a change in pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
EnricoMi commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679739264
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -75,23 +79,39 @@ class Observation(name: String) {
}
/**
- * Get the observed metrics. This waits for the observed dataset to finish its first action.
- * Only the result of the first action is available. Subsequent actions do not modify the result.
+ * (Scala-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific) Get the observed metrics. This waits for the observed dataset to finish
+ * its first action. Only the result of the first action is available. Subsequent actions do not
+ * modify the result.
+ *
+ * @return the observed metrics as a `java.util.Map[String, Object]`
+ * @throws InterruptedException interrupted while waiting
+ */
+ @throws[InterruptedException]
+ def getAsJava: java.util.Map[String, Object] = {
Review comment:
Java expects a `Map<String, ?>` with `Map[String, _]` and `Map<String, Object>` with `Map[String, AnyRef]`, so I'd go for the latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-887876643
**[Test build #141736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141736/testReport)** for PR 33545 at commit [`8d7aa97`](https://github.com/apache/spark/commit/8d7aa978269cb4f1a271dca3040bec6b6becbcb0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33545: [SPARK-36319][SQL][PySpark] Have Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33545:
URL: https://github.com/apache/spark/pull/33545#discussion_r679026047
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
Review comment:
```suggestion
dict
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific)
+ * Get the observed metrics. This waits for the observed dataset to finish its first action.
+ * Only the result of the first action is available. Subsequent actions do not modify the result.
+ *
Review comment:
```suggestion
* (Java-specific) Get the observed metrics. This waits for the observed dataset to finish
* its first action. Only the result of the first action is available. Subsequent actions do not
* modify the result.
*
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
Review comment:
```suggestion
* (Scala-specific) Get the observed metrics. This waits for the observed dataset to finish
* its first action. Only the result of the first action is available. Subsequent actions do not
* modify the result.
*
```
##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
##########
@@ -391,6 +390,60 @@ public void testGroupBy() {
Assert.assertEquals(asSet("1a#2", "3foobar#6", "5#10"), toSet(cogrouped.collectAsList()));
}
+ @Test
+ public void testObservation() {
+ // SPARK-34806: tests the Observation Java API and Dataset.observe(Observation, Column, Column*)
+ Observation namedObservation = new Observation("named");
+ Observation unnamedObservation = new Observation();
+
+ Dataset<Long> df = spark
+ .range(100)
+ .observe(
+ namedObservation,
+ min(col("id")).as("min_val"),
+ max(col("id")).as("max_val"),
+ sum(col("id")).as("sum_val"),
+ count(when(pmod(col("id"), lit(2)).$eq$eq$eq(0), 1)).as("num_even")
+ )
+ .observe(
+ unnamedObservation,
+ avg(col("id")).cast("int").as("avg_val")
+ );
+
+ df.collect();
+ Map<String, Object> namedMetrics = null;
+ Map<String, Object> unnamedMetrics = null;
+
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
Review comment:
```suggestion
namedMetrics = namedObservation.getAsJava();
unnamedMetrics = unnamedObservation.getAsJava();
```
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
the observed metrics
"""
assert self._jo is not None, 'call DataFrame.observe'
- jrow = self._jo.get()
- return self._to_row(jrow)
+ jmap = self._jo.getAsJavaMap()
+ # return a pure Python dict, not a py4j JavaMap
+ return {k: v for k, v in jmap.items()}
Review comment:
can you actually directly return `self._jo.getAsJavaMap()`. That becomes a plain dictionary IIRC.
##########
File path: python/pyspark/sql/observation.py
##########
@@ -112,12 +115,13 @@ def get(self):
Returns
-------
- :class:`Row`
+ :class:`Dict`
the observed metrics
"""
assert self._jo is not None, 'call DataFrame.observe'
- jrow = self._jo.get()
- return self._to_row(jrow)
+ jmap = self._jo.getAsJavaMap()
Review comment:
```suggestion
jmap = self._jo.getAsJava()
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific)
+ * Get the observed metrics. This waits for the observed dataset to finish its first action.
+ * Only the result of the first action is available. Subsequent actions do not modify the result.
+ *
+ * @return the observed metrics as a `java.util.Map[String, Object]`
+ * @throws InterruptedException interrupted while waiting
+ */
+ @throws[InterruptedException]
+ def getAsJavaMap: java.util.Map[String, Object] = {
Review comment:
```suggestion
def getAsJava: java.util.Map[String, Object] = {
```
##########
File path: sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
##########
@@ -391,6 +390,60 @@ public void testGroupBy() {
Assert.assertEquals(asSet("1a#2", "3foobar#6", "5#10"), toSet(cogrouped.collectAsList()));
}
+ @Test
+ public void testObservation() {
+ // SPARK-34806: tests the Observation Java API and Dataset.observe(Observation, Column, Column*)
+ Observation namedObservation = new Observation("named");
+ Observation unnamedObservation = new Observation();
+
+ Dataset<Long> df = spark
+ .range(100)
+ .observe(
+ namedObservation,
+ min(col("id")).as("min_val"),
+ max(col("id")).as("max_val"),
+ sum(col("id")).as("sum_val"),
+ count(when(pmod(col("id"), lit(2)).$eq$eq$eq(0), 1)).as("num_even")
+ )
+ .observe(
+ unnamedObservation,
+ avg(col("id")).cast("int").as("avg_val")
+ );
+
+ df.collect();
+ Map<String, Object> namedMetrics = null;
+ Map<String, Object> unnamedMetrics = null;
+
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
+ } catch (InterruptedException e) {
+ Assert.fail();
+ }
+ Map<String, Object> expectedNamedMetrics = new HashMap<String, Object>() {{
+ put("min_val", 0L);
+ put("max_val", 99L);
+ put("sum_val", 4950L);
+ put("num_even", 50L);
+ }};
+ Assert.assertEquals(expectedNamedMetrics, namedMetrics);
+
+ Map<String, Object> expectedUnnamedMetrics = new HashMap<String, Object>() {{
+ put("avg_val", 49);
+ }};
+ Assert.assertEquals(expectedUnnamedMetrics, unnamedMetrics);
+
+ // we can get the result multiple times
+ try {
+ namedMetrics = namedObservation.getAsJavaMap();
+ unnamedMetrics = unnamedObservation.getAsJavaMap();
Review comment:
```suggestion
namedMetrics = namedObservation.getAsJava();
unnamedMetrics = unnamedObservation.getAsJava();
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala
##########
@@ -78,20 +82,35 @@ class Observation(name: String) {
* Get the observed metrics. This waits for the observed dataset to finish its first action.
* Only the result of the first action is available. Subsequent actions do not modify the result.
*
- * @return the observed metrics as a [[Row]]
+ * @return the observed metrics as a `Map[String, Any]`
* @throws InterruptedException interrupted while waiting
*/
@throws[InterruptedException]
- def get: Row = {
+ def get: Map[String, Any] = {
synchronized {
// we need to loop as wait might return without us calling notify
// https://en.wikipedia.org/w/index.php?title=Spurious_wakeup&oldid=992601610
- while (this.row.isEmpty) {
+ while (this.metrics.isEmpty) {
wait()
}
}
- this.row.get
+ this.metrics.get
+ }
+
+ /**
+ * (Java-specific)
+ * Get the observed metrics. This waits for the observed dataset to finish its first action.
+ * Only the result of the first action is available. Subsequent actions do not modify the result.
+ *
+ * @return the observed metrics as a `java.util.Map[String, Object]`
+ * @throws InterruptedException interrupted while waiting
+ */
+ @throws[InterruptedException]
+ def getAsJavaMap: java.util.Map[String, Object] = {
Review comment:
Hmmm ... okay. this one might be one problem .. it requires to add another method which I actually would like to avoid ... but I guess it's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33545: [SPARK-36319][SQL][Python] Make Observation return Map instead of Row
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33545:
URL: https://github.com/apache/spark/pull/33545#issuecomment-889831897
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org