You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/03 08:03:45 UTC
[GitHub] [spark] beliefer opened a new pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
beliefer opened a new pull request #34474:
URL: https://github.com/apache/spark/pull/34474
### What changes were proposed in this pull request?
Currently,
```
val namedObservation = Observation("named")
val df = spark.range(100)
val observed_df = df.observe(
namedObservation, percentile_approx($"id", lit(0.5), lit(100)).as("percentile_approx_val"))
observed_df.collect()
namedObservation.get
```
throws exception as follows:
```
15:16:27.994 ERROR org.apache.spark.util.Utils: Exception encountered
java.io.NotSerializableException: org.apache.spark.sql.catalyst.expressions.aggregate.ApproximatePercentile$PercentileDigest
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.scheduler.DirectTaskResult.$anonfun$writeExternal$2(TaskResult.scala:55)
at org.apache.spark.scheduler.DirectTaskResult.$anonfun$writeExternal$2$adapted(TaskResult.scala:55)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.scheduler.DirectTaskResult.$anonfun$writeExternal$1(TaskResult.scala:55)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1434)
at org.apache.spark.scheduler.DirectTaskResult.writeExternal(TaskResult.scala:51)
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
This PR will fix the issue.
### Why are the changes needed?
Fix `NotSerializableException` when observe with `percentile_approx`.
### Does this PR introduce _any_ user-facing change?
Yes. `PercentileDigest` extends `Serializable`.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959610670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742691902
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AggregatingAccumulator.scala
##########
@@ -188,6 +197,18 @@ class AggregatingAccumulator private(
resultProjection(input)
}
+ override def withBufferSerialized(): AggregatingAccumulator = {
+ if (!isAtDriverSide) {
Review comment:
I think we can do `assert(!isAtDriverSide)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961209037
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49373/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
**[Test build #144873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144873/testReport)** for PR 34474 at commit [`1eacf82`](https://github.com/apache/spark/commit/1eacf82ae9ba0173b842a671cde8c82a4032a0cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961316859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961316859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960595922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r743337981
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -754,6 +754,17 @@ class DatasetSuite extends QueryTest
assert(err2.getMessage.contains("Name must not be empty"))
}
+ test("SPARK-37203: Fix NotSerializableException when observe with TypedImperativeAggregate") {
+ val namedObservation = Observation("named")
+
+ val df = spark.range(100)
+ val observed_df = df.observe(
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742401466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I probably forgot to add it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742699003
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AggregatingAccumulator.scala
##########
@@ -156,6 +156,15 @@ class AggregatingAccumulator private(
case agg: AggregatingAccumulator =>
val buffer = getOrCreateBuffer()
val otherBuffer = agg.buffer
+ // If AggregatingAccumulator runs on driver,
+ // we should deserialize all TypedImperativeAggregate.
+ if (isAtDriverSide) {
Review comment:
We don't need to add a new code. We can just change the existing code
```
while (i < typedImperatives.length) {
typedImperatives(i).mergeBuffersObjects(buffer, otherBuffer)
i += 1
}
```
to
```
if (atDriverSide) {
while (i < typedImperatives.length) {
// The input buffer stores serialized data
typedImperatives(i).merge(buffer, otherBuffer)
i += 1
}
} else {
while (i < typedImperatives.length) {
// The input buffer stores deserialized object
typedImperatives(i). mergeBuffersObjects(buffer, otherBuffer)
i += 1
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958784443
**[Test build #144875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144875/testReport)** for PR 34474 at commit [`a2f1fbb`](https://github.com/apache/spark/commit/a2f1fbbb2f74afefe802624700887ae89dfc7591).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742691619
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -621,6 +621,17 @@ abstract class TypedImperativeAggregate[T] extends ImperativeAggregate {
buffer(mutableAggBufferOffset) = serialize(getBufferObject(buffer))
}
+ /**
+ * In-place replaces SparkSQL internally supported underlying storage format (BinaryType),
+ * with the aggregation buffer object stored at buffer's index `mutableAggBufferOffset`.
+ *
+ * This is only called when AggregatingAccumulator running on driver, after the framework
+ * shuffle in aggregate buffers.
+ */
+ final def deserializeAggregateBufferInPlace(buffer: InternalRow): Unit = {
Review comment:
can't we just inline it? It seems overkill to introduce a new API in the base class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960636626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961639313
**[Test build #144913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144913/testReport)** for PR 34474 at commit [`acddbf3`](https://github.com/apache/spark/commit/acddbf34e66e70ad64d04b03d300e87049c9b73e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960595922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961599830
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961661517
> @beliefer can you update the PR description? `Yes. PercentileDigest extends Serializable.` this is wrong now.
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961708890
@cloud-fan Thanks for your hard work！@srowen @MaxGekk @hvanhovell ， thank you for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958803518
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959093308
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959093308
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144873/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959269981
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742632437
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
@MaxGekk @hvanhovell I updated the code and it will fix the issue for `AggregatingAccumulator` takes any `TypedImperativeAggregate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34474:
URL: https://github.com/apache/spark/pull/34474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742398645
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
Can we just add the proper logic to observe? Adding serialization will fix this particular problem, but it won't work for other aggregates.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I probably forgot to add it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742639262
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AggregatingAccumulator.scala
##########
@@ -185,9 +194,26 @@ class AggregatingAccumulator private(
// query execution).
createBuffer()
}
+ }
+
+ override def value: InternalRow = withSQLConf(false, InternalRow.empty) {
+ val input = getOrCreateTempBuffer()
resultProjection(input)
}
+ override def withBufferSerialized(): AggregatingAccumulator = {
+ if (!isAtDriverSide) {
+ val input = getOrCreateTempBuffer()
Review comment:
why do we need to create temp buffer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960557164
**[Test build #144898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144898/testReport)** for PR 34474 at commit [`6c2f71c`](https://github.com/apache/spark/commit/6c2f71c6c1ee27941499fafc0b1fad59b646caa2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961209037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961612684
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961685874
> > Yes. This PR let AggregatingAccumulator uses serialize and deserialize of TypedImperativeAggregate.
>
> Is it a user-facing change?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742960561
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -754,6 +754,17 @@ class DatasetSuite extends QueryTest
assert(err2.getMessage.contains("Name must not be empty"))
}
+ test("SPARK-37203: Fix NotSerializableException when observe with TypedImperativeAggregate") {
+ val namedObservation = Observation("named")
+
+ val df = spark.range(100)
+ val observed_df = df.observe(
Review comment:
can we test a DataFrame with no data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34474:
URL: https://github.com/apache/spark/pull/34474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961584437
**[Test build #144913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144913/testReport)** for PR 34474 at commit [`acddbf3`](https://github.com/apache/spark/commit/acddbf34e66e70ad64d04b03d300e87049c9b73e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961666147
> Yes. This PR let AggregatingAccumulator uses serialize and deserialize of TypedImperativeAggregate.
Is it a user-facing change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r743337981
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -754,6 +754,17 @@ class DatasetSuite extends QueryTest
assert(err2.getMessage.contains("Name must not be empty"))
}
+ test("SPARK-37203: Fix NotSerializableException when observe with TypedImperativeAggregate") {
+ val namedObservation = Observation("named")
+
+ val df = spark.range(100)
+ val observed_df = df.observe(
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961649178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960595922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961287930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959610670
Looking at the implementation, it's never meant to be serialized by Java or Kryo, but is serialized 'manually' and internally by Spark. I think it must be related to observe() here, that somehow this object gets caught up in what is returned? I don't know a lot about observe().
It's possible the answer is that observe() should do something different, but not sure.
We can make this Serializable, but then need to implement methods for the Java serializer that would reuse the existing serializer code. That seems plausible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960592038
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960693705
**[Test build #144898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144898/testReport)** for PR 34474 at commit [`6c2f71c`](https://github.com/apache/spark/commit/6c2f71c6c1ee27941499fafc0b1fad59b646caa2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961710058
thanks, merging to master/3.2/3.1!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961661517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961167333
**[Test build #144903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144903/testReport)** for PR 34474 at commit [`5e7afac`](https://github.com/apache/spark/commit/5e7afacde67bd1c943006250d3aebfbc72cdc5c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961272114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958852503
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742400130
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I agree if that's possible - I couldn't tell if this was something observe() should never have to pull back and does accidentally, or whether it is necessary to return metrics about percentile_approx somehow
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I agree if that's possible - I couldn't tell if this was something observe() should never have to pull back and does accidentally, or whether it is necessary to return metrics about percentile_approx somehow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742632437
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
@MaxGekk @hvanhovell @srowen I updated the code and it will fix the issue for `AggregatingAccumulator` takes any `TypedImperativeAggregate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960557164
**[Test build #144898 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144898/testReport)** for PR 34474 at commit [`6c2f71c`](https://github.com/apache/spark/commit/6c2f71c6c1ee27941499fafc0b1fad59b646caa2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961649178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960700466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742400130
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I agree if that's possible - I couldn't tell if this was something observe() should never have to pull back and does accidentally, or whether it is necessary to return metrics about percentile_approx somehow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960623031
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961661517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961649178
@beliefer can you update the PR description? `Yes. PercentileDigest extends Serializable.` this is wrong now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742690800
##########
File path: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala
##########
@@ -157,6 +157,9 @@ abstract class AccumulatorV2[IN, OUT] extends Serializable {
*/
def value: OUT
+ // We assume that serialization of AccumulatorV2 runs on executor is not necessary.
Review comment:
```
// Serialize the buffer of this accumulator before sending back this accumulator to the driver.
// By default this method does nothing.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961641653
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144913/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742632437
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
@hvanhovell I updated the code and it will fix the issue for `AggregatingAccumulator` takes any `TypedImperativeAggregate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r741980466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
hmm, there are special methods for (de-)serialization of `PercentileDigest`:
https://github.com/apache/spark/blob/08123a3795683238352e5bf55452de381349fdd9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L212-L218
Do you know why the methods are used instead of extending of `Serializable`?
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958773101
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959071536
**[Test build #144873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144873/testReport)** for PR 34474 at commit [`1eacf82`](https://github.com/apache/spark/commit/1eacf82ae9ba0173b842a671cde8c82a4032a0cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r741980466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
hmm, there are special methods for (de-)serialization of `PercentileDigest`:
https://github.com/apache/spark/blob/08123a3795683238352e5bf55452de381349fdd9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L212-L218
Do you know why the methods are used instead of extending of `Serializable`?
cc @cloud-fan
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
hmm, there are special methods for (de-)serialization of `PercentileDigest`:
https://github.com/apache/spark/blob/08123a3795683238352e5bf55452de381349fdd9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L212-L218
Do you know why the methods are used instead of extending of `Serializable`?
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958932529
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r741980466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
hmm, there are special methods for (de-)serialization of `PercentileDigest`:
https://github.com/apache/spark/blob/08123a3795683238352e5bf55452de381349fdd9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L212-L218
Do you know why the methods are used instead of extending of `Serializable`?
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742398645
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
Can we just add the proper logic to observe? Adding serialization will fix this particular problem, but it won't work for other aggregates.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I probably forgot to add it...
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
Can we just add the proper logic to observe? Adding serialization will fix this particular problem, but it won't work for other aggregates.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I probably forgot to add it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742694720
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -621,6 +621,17 @@ abstract class TypedImperativeAggregate[T] extends ImperativeAggregate {
buffer(mutableAggBufferOffset) = serialize(getBufferObject(buffer))
}
+ /**
+ * In-place replaces SparkSQL internally supported underlying storage format (BinaryType),
+ * with the aggregation buffer object stored at buffer's index `mutableAggBufferOffset`.
Review comment:
`stored at buffer's index mutableAggBufferOffset` describes the binary, should be put before `with the aggregation buffer object ...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960636626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49368/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742633954
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AggregatingAccumulator.scala
##########
@@ -156,6 +156,15 @@ class AggregatingAccumulator private(
case agg: AggregatingAccumulator =>
val buffer = getOrCreateBuffer()
val otherBuffer = agg.buffer
+ // If AggregatingAccumulator runs on driver,
+ // we should deserialize all TypedImperativeAggregate.
Review comment:
shall we deserialize in `AccumulatorV2.readObject`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958947626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959250130
**[Test build #144875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144875/testReport)** for PR 34474 at commit [`a2f1fbb`](https://github.com/apache/spark/commit/a2f1fbbb2f74afefe802624700887ae89dfc7591).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958784443
**[Test build #144875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144875/testReport)** for PR 34474 at commit [`a2f1fbb`](https://github.com/apache/spark/commit/a2f1fbbb2f74afefe802624700887ae89dfc7591).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r743337981
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
##########
@@ -754,6 +754,17 @@ class DatasetSuite extends QueryTest
assert(err2.getMessage.contains("Name must not be empty"))
}
+ test("SPARK-37203: Fix NotSerializableException when observe with TypedImperativeAggregate") {
+ val namedObservation = Observation("named")
+
+ val df = spark.range(100)
+ val observed_df = df.observe(
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742398645
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
Can we just add the proper logic to observe? Adding serialization will fix this particular problem, but it won't work for other aggregates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961246341
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49373/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961272114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49373/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742400130
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
##########
@@ -226,7 +226,7 @@ object ApproximatePercentile {
*
* @param summaries underlying probabilistic data structure [[QuantileSummaries]].
*/
- class PercentileDigest(private var summaries: QuantileSummaries) {
+ class PercentileDigest(private var summaries: QuantileSummaries) extends Serializable {
Review comment:
I agree if that's possible - I couldn't tell if this was something observe() should never have to pull back and does accidentally, or whether it is necessary to return metrics about percentile_approx somehow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960665616
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742633954
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AggregatingAccumulator.scala
##########
@@ -156,6 +156,15 @@ class AggregatingAccumulator private(
case agg: AggregatingAccumulator =>
val buffer = getOrCreateBuffer()
val otherBuffer = agg.buffer
+ // If AggregatingAccumulator runs on driver,
+ // we should deserialize all TypedImperativeAggregate.
Review comment:
shall we deserialize in `AccumulatorV2.readObject`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960595922
**[Test build #144899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144899/testReport)** for PR 34474 at commit [`84f9657`](https://github.com/apache/spark/commit/84f96570917e646efdf196d37fb9a68eb2e94309).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34474:
URL: https://github.com/apache/spark/pull/34474#discussion_r742695249
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -621,6 +621,17 @@ abstract class TypedImperativeAggregate[T] extends ImperativeAggregate {
buffer(mutableAggBufferOffset) = serialize(getBufferObject(buffer))
}
+ /**
+ * In-place replaces SparkSQL internally supported underlying storage format (BinaryType),
+ * with the aggregation buffer object stored at buffer's index `mutableAggBufferOffset`.
+ *
+ * This is only called when AggregatingAccumulator running on driver, after the framework
+ * shuffle in aggregate buffers.
Review comment:
This is nothing to do with shuffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-960633539
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49369/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961287930
**[Test build #144899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144899/testReport)** for PR 34474 at commit [`84f9657`](https://github.com/apache/spark/commit/84f96570917e646efdf196d37fb9a68eb2e94309).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961584437
**[Test build #144913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144913/testReport)** for PR 34474 at commit [`acddbf3`](https://github.com/apache/spark/commit/acddbf34e66e70ad64d04b03d300e87049c9b73e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961623718
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34474:
URL: https://github.com/apache/spark/pull/34474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with TypedImperativeAggregate
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-961299333
**[Test build #144903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144903/testReport)** for PR 34474 at commit [`5e7afac`](https://github.com/apache/spark/commit/5e7afacde67bd1c943006250d3aebfbc72cdc5c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958947626
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49345/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959269981
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958863771
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49343/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-958743880
**[Test build #144873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144873/testReport)** for PR 34474 at commit [`1eacf82`](https://github.com/apache/spark/commit/1eacf82ae9ba0173b842a671cde8c82a4032a0cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #34474: [SPARK-37203][SQL] Fix NotSerializableException when observe with percentile_approx
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #34474:
URL: https://github.com/apache/spark/pull/34474#issuecomment-959610670
Looking at the implementation, it's never meant to be serialized by Java or Kryo, but is serialized 'manually' and internally by Spark. I think it must be related to observe() here, that somehow this object gets caught up in what is returned? I don't know a lot about observe().
It's possible the answer is that observe() should do something different, but not sure.
We can make this Serializable, but then need to implement methods for the Java serializer that would reuse the existing serializer code. That seems plausible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org