You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yhuai <gi...@git.apache.org> on 2016/07/18 18:56:08 UTC
[GitHub] spark pull request #14249: [SPARK-16515][SQL]set default record reader and w...
GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/14249
[SPARK-16515][SQL]set default record reader and writer for script transformation
## What changes were proposed in this pull request?
In ScriptInputOutputSchema, we read default RecordReader and RecordWriter from conf. Since Spark 2.0 has deleted those config keys from hive conf, we have to set default reader/writer class name by ourselves. Otherwise we will get None for LazySimpleSerde, the data written would not be able to read by script. The test case added worked fine with previous version of Spark, but would fail now.
## How was this patch tested?
added a test case in SQLQuerySuite.
Closes #14169
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark scriptTransformation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14249.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14249
----
commit e4c7e027e9216bad42ac85916de05e26163452ea
Author: Daoyuan Wang <da...@intel.com>
Date: 2016-07-13T10:29:48Z
set default record reader and writer in script transformation
commit 8048732c56bd43dd0ca175a0a047a495ab06fcea
Author: Daoyuan Wang <da...@intel.com>
Date: 2016-07-18T07:45:28Z
address comment
commit 814f67926038ef136efa12b8e41d8c252411b9c9
Author: Daoyuan Wang <da...@intel.com>
Date: 2016-07-18T07:45:56Z
Update SparkSqlParser.scala
commit 0edfed48a1e6a438a18d488404b900a51475a2d5
Author: Daoyuan Wang <da...@intel.com>
Date: 2016-07-18T07:46:56Z
Update SparkSqlParser.scala
commit e999f2c381fbd7c25695b16a38b97a9518e107fc
Author: Yin Huai <yh...@databricks.com>
Date: 2016-07-18T18:54:55Z
Update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14249
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14249
**[Test build #62479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62479/consoleFull)** for PR 14249 at commit [`e999f2c`](https://github.com/apache/spark/commit/e999f2c381fbd7c25695b16a38b97a9518e107fc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14249
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62479/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14249
**[Test build #62479 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62479/consoleFull)** for PR 14249 at commit [`e999f2c`](https://github.com/apache/spark/commit/e999f2c381fbd7c25695b16a38b97a9518e107fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14249: [SPARK-16515][SQL]set default record reader and w...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14249
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14249
I am merging this PR to master and branch 2.0.
Thanks @adrian-wang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14249: [SPARK-16515][SQL]set default record reader and w...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/14249#discussion_r71227856
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1329,7 +1332,7 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
// SPARK-10310: Special cases LazySimpleSerDe
val recordHandler = if (name == "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe") {
- Try(conf.getConfString(configKey)).toOption
+ Option(conf.getConfString(configKey, defaultConfigValue))
--- End diff --
This Option is used to turn the returned handler to the type that is expected by ScriptInputOutputSchema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org