You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2017/09/14 17:20:55 UTC
[GitHub] spark pull request #19237: [SPARK-21987][SQL] fix a compatibility issue of s...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19237
[SPARK-21987][SQL] fix a compatibility issue of sql event logs
## What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/18600 we removed the `metadata` field from `SparkPlanInfo`. This causes a problem when we replay event logs that are generated by older Spark versions.
## How was this patch tested?
a regression test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark event
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19237.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19237
----
commit 414457cfeca4066bff2491ea81851c053cb9c6c7
Author: Wenchen Fan <we...@databricks.com>
Date: 2017-09-14T17:14:42Z
fix json protocol
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19237: [SPARK-21987][SQL] fix a compatibility issue of s...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19237
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19237: [SPARK-21987][SQL] fix a compatibility issue of s...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19237#discussion_r138963733
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLJsonProtocolSuite.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import org.json4s.jackson.JsonMethods.parse
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart
+import org.apache.spark.util.JsonProtocol
+
+class SQLJsonProtocolSuite extends SparkFunSuite {
+
+ test("SparkPlanGraph backward compatibility: metadata") {
+ val reconstructedEvent = JsonProtocol.sparkEventFromJson(parse(SQLExecutionStartJsonString))
+ val expectedEvent = SparkListenerSQLExecutionStart(0, "test desc", "test detail", "test plan",
+ new SparkPlanInfo("TestNode", "test string", Nil, Nil), 0)
+ assert(reconstructedEvent == expectedEvent)
+ }
+
+ private val SQLExecutionStartJsonString =
--- End diff --
Declare this inside the test since it's only used there? (Also would make it easier to match the test to the data if more tests are added.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19237
cc @vanzin @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19237
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19237
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81791/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19237
**[Test build #81791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81791/testReport)** for PR 19237 at commit [`414457c`](https://github.com/apache/spark/commit/414457cfeca4066bff2491ea81851c053cb9c6c7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19237
**[Test build #81799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81799/testReport)** for PR 19237 at commit [`93cacba`](https://github.com/apache/spark/commit/93cacbabfe72a1c7dba20d00612757ae6e11f854).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19237
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19237
**[Test build #81791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81791/testReport)** for PR 19237 at commit [`414457c`](https://github.com/apache/spark/commit/414457cfeca4066bff2491ea81851c053cb9c6c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19237
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19237
**[Test build #81799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81799/testReport)** for PR 19237 at commit [`93cacba`](https://github.com/apache/spark/commit/93cacbabfe72a1c7dba20d00612757ae6e11f854).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19237: [SPARK-21987][SQL] fix a compatibility issue of sql even...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19237
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81799/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19237: [SPARK-21987][SQL] fix a compatibility issue of s...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19237#discussion_r138963502
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala ---
@@ -26,6 +28,7 @@ import org.apache.spark.sql.execution.metric.SQLMetricInfo
* Stores information about a SQL SparkPlan.
*/
@DeveloperApi
+@JsonIgnoreProperties(Array("metadata")) // The metadata filed was removed in Spark 2.3.
--- End diff --
field
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org