[jira] [Comment Edited] (SPARK-23056) parse_url regression when switched to using java.net.URI instead of java.net.URL

["Yash Datta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-23056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324056#comment-16324056 ] 

Yash Datta edited comment on SPARK-23056 at 1/12/18 2:52 PM:
-------------------------------------------------------------

We have production use case with many different IRIs in Japanese.
https://www.w3.org/International/articles/idn-and-iri/

Agreed that we should not be re-introducing performance bottlenecks, that is why did not submit a patch reverting to the use of URL instead of URI, but at-least for us this is a valid use case.

If this is not a generic enough problem to be solved, we can close it, but I disagree.


was (Author: saucam):
We have production use case with many different IRIs in Japanese.
https://www.w3.org/International/articles/idn-and-iri/

Agreed that we should not be re-introducing performance bottlenecks, that is why did not submit a patch reverting to the use of URL instead of URI, but at-least for us this is a valid use case.

> parse_url regression when switched to using java.net.URI instead of java.net.URL
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-23056
>                 URL: https://issues.apache.org/jira/browse/SPARK-23056
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.3, 2.2.2, 2.3.0
>            Reporter: Yash Datta
>              Labels: regression
>
> When using internationalized Domains in the urls like:
> {code:java}
> val url = "http://правительство.рф"
> {code}
> The parse_url returns null, but works fine when using the hive 's version of parse_url
> On digging further, found that the difference is in below call in spark:
> {code:java}
> private def getUrl(url: UTF8String): URI = {
>   try {
>     new URI(url.toString)
>   } catch {
>     case e: URISyntaxException => null
>   }
> }
> {code}
> while hive uses java.net.URL:
> {code:java}
> url = new URL(urlStr)
> {code}
> Sure enough, this simple test demonstrates URL works but URI does not in this case:
> {code:java}
> val url = "http://правительство.рф"
> val uriHost = new URI(url).getHost
> val urlHost = new URL(url).getHost
> println(s"uriHost = $uriHost")     // prints uriHost = null
> println(s"urlHost = $urlHost") // prints urlHost = правительство.рф  
> {code}
> To reproduce the problem on spark-sql:
> {code:java}
> spark-sql> select parse_url('http://千夏ともか.test', 'HOST');
> {code}
> returns NULL
> This problem was introduced by
> <https://issues.apache.org/jira/browse/SPARK-16826> which is designed to
> improve the performance of PARSE_URL().
> The same issue exists in the following SQL:
> {code:java}
> SELECT PARSE_URL('http://stanzhai.site?p=["abc"]', 'QUERY', 'p')
> {code}
> // return null in Spark 2.1+
> // return ["abc"] less than Spark 2.1
> ```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org