You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by YanTangZhai <gi...@git.apache.org> on 2014/10/20 15:07:27 UTC
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/2857
[SPARK-4009][SQL]HiveTableScan should use makeRDDForTable instead of makeRDDForPartitionedTable for partitioned table when partitionPruningPred is None
HiveTableScan should use makeRDDForTable instead of makeRDDForPartitionedTable for partitioned table when partitionPruningPred is None.
If a table has many partitions for example more than 20 thousands while it has a few data for example less than 512MB, some sql querying the table will produce more than 20000 RDDs. The job would submit failed with exception: java stack overflow.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YanTangZhai/spark SPARK-4009
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2857.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2857
----
commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-06T13:07:08Z
Merge pull request #1 from apache/master
update
commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-20T13:14:08Z
Merge pull request #3 from apache/master
Update
commit 8a0010691b669495b4c327cf83124cabb7da1405
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-12T06:54:58Z
Merge pull request #6 from apache/master
Update
commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-09-16T12:03:22Z
Merge pull request #7 from apache/master
Update
commit 76d40277d51f709247df1d3734093bf2c047737d
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-10-20T12:52:22Z
Merge pull request #8 from apache/master
update
commit be7882ce16911d018571fa46c1a175d063bdfd03
Author: yantangzhai <ty...@163.com>
Date: 2014-10-20T13:05:44Z
[SPARK-4009][SQL]HiveTableScan should use makeRDDForTable instead of makeRDDForPartitionedTable for partitioned table when partitionPruningPred is None
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai closed the pull request at:
https://github.com/apache/spark/pull/2857
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59758705
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21917/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59751379
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21916/consoleFull) for PR 2857 at commit [`be7882c`](https://github.com/apache/spark/commit/be7882ce16911d018571fa46c1a175d063bdfd03).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59816505
This isn't a valid change since different partitions of a table can have different serializers. As a result I think we should close this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59755633
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21917/consoleFull) for PR 2857 at commit [`db0ce73`](https://github.com/apache/spark/commit/db0ce732e51d5813609f80722c20147b7c33bd23).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59751519
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21916/consoleFull) for PR 2857 at commit [`be7882c`](https://github.com/apache/spark/commit/be7882ce16911d018571fa46c1a175d063bdfd03).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59751523
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21916/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59758697
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21917/consoleFull) for PR 2857 at commit [`db0ce73`](https://github.com/apache/spark/commit/db0ce732e51d5813609f80722c20147b7c33bd23).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class JavaFutureActionWrapper[S, T](futureAction: FutureAction[S], converter: S => T)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4009][SQL]HiveTableScan should use make...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on the pull request:
https://github.com/apache/spark/pull/2857#issuecomment-59915528
@marmbrus Thanks. Please disregard it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org