You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gengliangwang <gi...@git.apache.org> on 2018/08/17 10:46:07 UTC
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22133
[SPARK-25129][SQL]Make the mapping of com.databricks.spark.avro to built-in module configurable
## What changes were proposed in this pull request?
In https://issues.apache.org/jira/browse/SPARK-24924, the data source provider com.databricks.spark.avro is mapped to the new package org.apache.spark.sql.avro .
As per the discussion in the [Jira](https://issues.apache.org/jira/browse/SPARK-24924) and PR #22119, we should make the mapping configurable.
## How was this patch tested?
Unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark configurable_avro_mapping
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22133.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22133
----
commit 0d6ba2a2483f58176ca26dfe8a3accd6bc49e8e1
Author: Gengliang Wang <ge...@...>
Date: 2018-08-17T10:30:27Z
make databricks avro backward compatibility configurable
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211282862
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -637,6 +638,17 @@ object DataSource extends Logging {
"Hive built-in ORC data source must be used with Hive support enabled. " +
"Please use the native ORC data source by setting 'spark.sql.orc.impl' to " +
"'native'")
+ } else if (provider1.toLowerCase(Locale.ROOT) == "avro" ||
+ provider1 == "com.databricks.spark.avro" ||
+ provider1 == "org.apache.spark.sql.avro") {
+ throw new AnalysisException(
+ s"Failed to find data source: $provider1. Avro is built-in data source since " +
+ "Spark 2.4. Please deploy the application as per " +
+ s"$latestDocsURL/avro-data-source-guide.html#deploying")
--- End diff --
@tgravescs Make sense, thank you.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94996/testReport)** for PR 22133 at commit [`e57b232`](https://github.com/apache/spark/commit/e57b232ec8e36ea107ce103e5cdb6efaa0756c40).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94950/testReport)** for PR 22133 at commit [`be69aec`](https://github.com/apache/spark/commit/be69aec8f7d24f9af49edc8a70961e8991765287).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211259701
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1474,6 +1474,13 @@ object SQLConf {
.checkValues((1 to 9).toSet + Deflater.DEFAULT_COMPRESSION)
.createWithDefault(Deflater.DEFAULT_COMPRESSION)
+ val LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED =
+ buildConf("spark.sql.legacy.replaceDatabricksSparkAvro.enabled")
+ .doc("If it is set to true, the data source provider com.databricks.spark.avro is mapped " +
+ "to the built-in Avro data source module for backward compatibility.")
--- End diff --
do we want to give more details here about being for hive table provider compatibility?
I think it would also be nice if we put more details about the compatibility in the avro doc, either in this pr or in the other doc one.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94885 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94885/testReport)** for PR 22133 at commit [`0d6ba2a`](https://github.com/apache/spark/commit/0d6ba2a2483f58176ca26dfe8a3accd6bc49e8e1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94960/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211482461
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
This is the link for the latest doc. I think it should be ok.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211482149
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -593,7 +592,6 @@ object DataSource extends Logging {
"org.apache.spark.ml.source.libsvm.DefaultSource" -> libsvm,
"org.apache.spark.ml.source.libsvm" -> libsvm,
"com.databricks.spark.csv" -> csv,
- "com.databricks.spark.avro" -> avro,
--- End diff --
@HyukjinKwon I did add it in the `backwardCompatibilityMap` at first. But later on I find that the configuration won't be effective in run time, since the `backwardCompatibilityMap` is a `val`. (We can change `backwardCompatibilityMap` to method to resolve that.) Also the code looks ugly.
```
val ret = Map(...)
if(...) {
ret + k -> v
} else {
ret
}
// it would be worse if we have more configurations.
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94960/testReport)** for PR 22133 at commit [`d617db0`](https://github.com/apache/spark/commit/d617db043ffe3841ec416bce09bfc261c0f32bc7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94950/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #95024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95024/testReport)** for PR 22133 at commit [`b22834f`](https://github.com/apache/spark/commit/b22834fdb3eaf9e4bd166ea4397326cd44815e3f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94885/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2355/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211561656
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
Ah, make sense. Thank you!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r210880791
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1474,6 +1474,12 @@ object SQLConf {
.checkValues((1 to 9).toSet + Deflater.DEFAULT_COMPRESSION)
.createWithDefault(Deflater.DEFAULT_COMPRESSION)
+ val ENABLE_AVRO_BACKWARD_COMPATIBILITY = buildConf("spark.sql.avro.backwardCompatibility")
--- End diff --
Thank you for working on this. I'm thinking if we can provide this in a general way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211485248
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
The doc will be like
https://github.com/apache/spark/pull/22121/files#diff-acdddc6cbd45ccd226bf151564b9cc40R11
It is about loading the module with `--package`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211260375
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -637,6 +638,17 @@ object DataSource extends Logging {
"Hive built-in ORC data source must be used with Hive support enabled. " +
"Please use the native ORC data source by setting 'spark.sql.orc.impl' to " +
"'native'")
+ } else if (provider1.toLowerCase(Locale.ROOT) == "avro" ||
+ provider1 == "com.databricks.spark.avro" ||
+ provider1 == "org.apache.spark.sql.avro") {
+ throw new AnalysisException(
+ s"Failed to find data source: $provider1. Avro is built-in data source since " +
+ "Spark 2.4. Please deploy the application as per " +
+ s"$latestDocsURL/avro-data-source-guide.html#deploying")
--- End diff --
perhaps we should say built-in but external module. As if its built-in I would expect it to automatically be there.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211482547
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -593,7 +592,6 @@ object DataSource extends Logging {
"org.apache.spark.ml.source.libsvm.DefaultSource" -> libsvm,
"org.apache.spark.ml.source.libsvm" -> libsvm,
"com.databricks.spark.csv" -> csv,
- "com.databricks.spark.avro" -> avro,
--- End diff --
Ah okie makes sense if there's a reason.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22133
Seems fine otherwise.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95001/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94940/testReport)** for PR 22133 at commit [`be69aec`](https://github.com/apache/spark/commit/be69aec8f7d24f9af49edc8a70961e8991765287).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22133
+1
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94940/testReport)** for PR 22133 at commit [`be69aec`](https://github.com/apache/spark/commit/be69aec8f7d24f9af49edc8a70961e8991765287).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22133
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2270/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2375/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211537581
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
But what if we happen to have more information specific to newer versions ..? For example, we could happen to have different group name rule, etc. in the future. Avoiding pointing out the latest shouldn't be too difficult .. You could just say "Please refer deployment section in Apache Avro data source guide.". if we happen to have other new changes specific to newer versions there, we should go down and fix the links in all the branches strictly in theory.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211460825
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -593,7 +592,6 @@ object DataSource extends Logging {
"org.apache.spark.ml.source.libsvm.DefaultSource" -> libsvm,
"org.apache.spark.ml.source.libsvm" -> libsvm,
"com.databricks.spark.csv" -> csv,
- "com.databricks.spark.avro" -> avro,
--- End diff --
@gengliangwang, not a big deal but how about adding the entry at 618 here conditionally since this is called backward compatibility map?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22133
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2324/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211460921
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
I would actually avoid to leave the explicit doc link because we will have to fix it for every release. Just prose should be good enough.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #95024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95024/testReport)** for PR 22133 at commit [`b22834f`](https://github.com/apache/spark/commit/b22834fdb3eaf9e4bd166ea4397326cd44815e3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94894/testReport)** for PR 22133 at commit [`400a7f3`](https://github.com/apache/spark/commit/400a7f32b16c4cdcaadf7461a9c70278d3640d16).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94960/testReport)** for PR 22133 at commit [`d617db0`](https://github.com/apache/spark/commit/d617db043ffe3841ec416bce09bfc261c0f32bc7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2310/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2360/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94885 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94885/testReport)** for PR 22133 at commit [`0d6ba2a`](https://github.com/apache/spark/commit/0d6ba2a2483f58176ca26dfe8a3accd6bc49e8e1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2316/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94940/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #95001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95001/testReport)** for PR 22133 at commit [`e57b232`](https://github.com/apache/spark/commit/e57b232ec8e36ea107ce103e5cdb6efaa0756c40).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94996/testReport)** for PR 22133 at commit [`e57b232`](https://github.com/apache/spark/commit/e57b232ec8e36ea107ce103e5cdb6efaa0756c40).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211289008
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1474,6 +1474,13 @@ object SQLConf {
.checkValues((1 to 9).toSet + Deflater.DEFAULT_COMPRESSION)
.createWithDefault(Deflater.DEFAULT_COMPRESSION)
+ val LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED =
+ buildConf("spark.sql.legacy.replaceDatabricksSparkAvro.enabled")
+ .doc("If it is set to true, the data source provider com.databricks.spark.avro is mapped " +
+ "to the built-in Avro data source module for backward compatibility.")
--- End diff --
Thanks, I will do it in https://github.com/apache/spark/pull/22121
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94894/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94996/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22133
retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/22133
@tgravescs @dongjoon-hyun @HyukjinKwon @cloud-fan
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2278/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r210873839
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -637,6 +638,17 @@ object DataSource extends Logging {
"Hive built-in ORC data source must be used with Hive support enabled. " +
"Please use the native ORC data source by setting 'spark.sql.orc.impl' to " +
"'native'")
+ } else if (provider1.toLowerCase(Locale.ROOT) == "avro" ||
+ provider1 == "com.databricks.spark.avro" ||
+ provider1 == "org.apache.spark.sql.avro") {
+ throw new AnalysisException(
+ s"Failed to find data source: $provider1. Avro is built-in data source since " +
+ "Spark 2.4. Please deploy the application as per " +
+ s"$latestDocsURL/avro-data-source-guide.html#deploying")
--- End diff --
Please merge https://github.com/apache/spark/pull/22121 before this one is merged.
But first we need to have agreement on the configuration name, since it is also mentioned in the documentation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r211482746
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -626,6 +626,7 @@ object DataSource extends Logging {
serviceLoader.asScala.filter(_.shortName().equalsIgnoreCase(provider1)).toList match {
// the provider format did not match any given registered aliases
case Nil =>
+ val latestDocsURL = "https://spark.apache.org/docs/latest"
--- End diff --
I mean, if we happen to have Spark 3.0.0 then this link will be stale in 2.4.0.. no?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94894/testReport)** for PR 22133 at commit [`400a7f3`](https://github.com/apache/spark/commit/400a7f32b16c4cdcaadf7461a9c70278d3640d16).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95024/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22133
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #94950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94950/testReport)** for PR 22133 at commit [`be69aec`](https://github.com/apache/spark/commit/be69aec8f7d24f9af49edc8a70961e8991765287).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22133: [SPARK-25129][SQL]Make the mapping of com.databricks.spa...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22133
**[Test build #95001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95001/testReport)** for PR 22133 at commit [`e57b232`](https://github.com/apache/spark/commit/e57b232ec8e36ea107ce103e5cdb6efaa0756c40).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22133: [SPARK-25129][SQL]Make the mapping of com.databri...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22133#discussion_r210959550
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
@@ -637,6 +638,17 @@ object DataSource extends Logging {
"Hive built-in ORC data source must be used with Hive support enabled. " +
"Please use the native ORC data source by setting 'spark.sql.orc.impl' to " +
"'native'")
+ } else if (provider1.toLowerCase(Locale.ROOT) == "avro" ||
+ provider1 == "com.databricks.spark.avro" ||
+ provider1 == "org.apache.spark.sql.avro") {
+ throw new AnalysisException(
+ s"Failed to find data source: $provider1. Avro is built-in data source since " +
+ "Spark 2.4. Please deploy the application as per " +
+ s"$latestDocsURL/avro-data-source-guide.html#deploying")
--- End diff --
We can update the message later. No need to be blocked by that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org