You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rajeshbalamohan <gi...@git.apache.org> on 2016/04/20 03:08:05 UTC
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
GitHub user rajeshbalamohan opened a pull request:
https://github.com/apache/spark/pull/12514
[SPARK-14521][SQL] StackOverflowError in Kryo when executing TPC-DS Q…
## What changes were proposed in this pull request?
Observed stackOverflowError in Kryo when executing TPC-DS Query27. Spark thrift server disables kryo reference tracking (if not specified in conf). When "spark.kryo.referenceTracking" is set to true explicitly in spark-defaults.conf, query executes successfully. Recent changes HashedRelation could have introduced loops which would need "spark.kryo.referenceTracking=true" in spark-thrift server. This PR addresses this by setting referenceTracking to true in SparkSQLEnv
## How was this patch tested?
Manually running TPC-DS queries at 200 GB scale in multi node cluster. Also ran org.apache.spark.sql.hive.execution.HiveCompatibilitySuite,org.apache.spark.sql.hive.execution.HiveQuerySuite,org.apache.spark.sql.hive.execution.PruningSuite,org.apache.spark.sql.hive/CachedTableSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rajeshbalamohan/spark SPARK-14521
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12514.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12514
----
commit 59875f424aaf60aa90ca5a1006df8b5e20d4f83a
Author: Rajesh Balamohan <rb...@apache.org>
Date: 2016-04-20T00:59:32Z
[SPARK-14521][SQL] StackOverflowError in Kryo when executing TPC-DS Query27
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212323253
(Or if for whatever reason you do change this, update the docs to note that the default is `true` everywhere)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan closed the pull request at:
https://github.com/apache/spark/pull/12514
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212191656
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-213215983
sure @yzhou2001 . Please go ahead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-214668848
@rajeshbalamohan please close this PR then
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212392651
Sure, will check on removing the circular reference. Took the reference tracking approach, as it is enabled by default with Spark's KryoSerializer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212576720
Yeah, modifying `HashedRelation` is probably the better approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212266446
Thanks for doing that.
I think maybe the right fix is to remove the circular reference, rather than just turning on reference tracking in Kryo? Otherwise if users use Spark outside the server and have reference tracking off, they will run into the same issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by yzhou2001 <gi...@git.apache.org>.
Github user yzhou2001 commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-213079734
The real culprit seems to be the "TaskMemoryManager" member of LongToUnsafeRowMap inside HashedRelation.scala. It should not be serialized and passed around, IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212213449
@rajeshbalamohan from now on, can you have the full title for your prs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by yzhou2001 <gi...@git.apache.org>.
Github user yzhou2001 commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-213209165
@rajeshbalamohan if you don't mind, I'll create a PR based on https://github.com/yzhou2001/spark/commit/6ba1a74aa4a35fa6d2d855d889afd4a3b688a27c
Query 27 still fail because of a NPE thrown from Kryo serializer, I believe, due to SPARK-14752 as you reported.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212574013
Reference tracking has a performance hit, and in many applications they are disabled explicitly. (e.g. graphx)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by rajeshbalamohan <gi...@git.apache.org>.
Github user rajeshbalamohan commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212191093
\cc @JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14521][SQL] StackOverflowError in Kryo ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12514#issuecomment-212572595
Yeah, as @rajeshbalamohan points out it looks like `KryoSerializer`'s default is to enable reference tracking and that the SQL default here is actually inconsistent with it: https://github.com/apache/spark/blob/cb8ea9e1f34b9af287b3d10e47f24de4307c63ba/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala#L71
Given this, the change here seems reasonable to me because it's actually bringing the ThriftServer's behavior more in line with the defaults in Spark Core. I wonder whether there's a cleaner way of doing this, though, perhaps by setting `spark.kryo.referenceTracking` only if `maybeKryoReferenceTracking` is defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org