You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bo Yang (JIRA)" <ji...@apache.org> on 2018/10/09 22:28:00 UTC
[jira] [Updated] (SPARK-25694) URL.setURLStreamHandlerFactory
causing incompatible HttpURLConnection
[ https://issues.apache.org/jira/browse/SPARK-25694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bo Yang updated SPARK-25694:
----------------------------
Description:
URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlConnection object, which is not compatible with HttpURLConnection. This will cause exception when using some third party http library (e.g. scalaj.http).
The following code in Spark 2.3.0 introduced the issue:
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
{quote}object SharedState extends Logging {
...
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
...
}
{quote}
Example exception when using scalaj.http in Spark:
StackTrace: scala.MatchError: org.apache.hadoop.fs.FsUrlConnection:http://wwww.example.com (of class org.apache.hadoop.fs.FsUrlConnection)
at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:343)
at scalaj.http.HttpRequest.exec(Http.scala:335)
at scalaj.http.HttpRequest.asString(Http.scala:455)
> URL.setURLStreamHandlerFactory causing incompatible HttpURLConnection
> ---------------------------------------------------------------------
>
> Key: SPARK-25694
> URL: https://issues.apache.org/jira/browse/SPARK-25694
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.3.0, 2.3.1, 2.3.2
> Reporter: Bo Yang
> Priority: Major
>
> URL.setURLStreamHandlerFactory() in SharedState causes URL.openConnection() returns FsUrlConnection object, which is not compatible with HttpURLConnection. This will cause exception when using some third party http library (e.g. scalaj.http).
> The following code in Spark 2.3.0 introduced the issue:
> sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
> {quote}object SharedState extends Logging {
> ...
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
> ...
> }
> {quote}
>
> Example exception when using scalaj.http in Spark:
> StackTrace: scala.MatchError: org.apache.hadoop.fs.FsUrlConnection:http://wwww.example.com (of class org.apache.hadoop.fs.FsUrlConnection)
> at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:343)
> at scalaj.http.HttpRequest.exec(Http.scala:335)
> at scalaj.http.HttpRequest.asString(Http.scala:455)
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org