You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yangw1234 <gi...@git.apache.org> on 2016/09/14 15:12:03 UTC
[GitHub] spark pull request #15096: [SPARK-17537] [SQL] Reading parquet schema from d...
GitHub user yangw1234 opened a pull request:
https://github.com/apache/spark/pull/15096
[SPARK-17537] [SQL] Reading parquet schema from driver directly when there is only one file to touch
## What changes were proposed in this pull request?
`spark.read.parquet("parquet/dir")` would issue a spark job to read parquet schema. When `spark.sql.parquet.mergeSchema` are set to false (the default value), there is often only one file to read, so there is no need to issue a spark job to do it. In this case, we can read it from driver directly instead of issuing a spark job. This could reduce the infer schema latency from several hundreds milliseconds to around ten milliseconds in my environment.
## How was this patch tested?
manually tested
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yangw1234/spark mergeSchema_2.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15096.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15096
----
commit 91081bccf4326e8261dd41035c199fbd8f12c69d
Author: wangyang <wa...@haizhi.com>
Date: 2016-09-14T13:12:24Z
opt merge schema
commit 9b610c22e8fedbf61b98943ca9a8b3ba08320c94
Author: wangyang <wa...@haizhi.com>
Date: 2016-09-14T13:20:49Z
fix typo
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15096: [SPARK-17537] [SQL] Reading parquet schema from driver d...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 commented on the issue:
https://github.com/apache/spark/pull/15096
Yes, this is a duplicate, closing it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15096: [SPARK-17537] [SQL] Reading parquet schema from d...
Posted by yangw1234 <gi...@git.apache.org>.
Github user yangw1234 closed the pull request at:
https://github.com/apache/spark/pull/15096
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15096: [SPARK-17537] [SQL] Reading parquet schema from driver d...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15096
Yes please close this as a duplicate
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15096: [SPARK-17537] [SQL] Reading parquet schema from driver d...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15096
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15096: [SPARK-17537] [SQL] Reading parquet schema from driver d...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/15096
Is this a duplicate of https://github.com/apache/spark/pull/14660? I think the PR tries to deal with the exact same case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org