You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/21 07:12:34 UTC

[GitHub] [spark] wangyum opened a new pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

wangyum opened a new pull request #28883:
URL: https://github.com/apache/spark/pull/28883


   ### What changes were proposed in this pull request?
   
   This pr add Support `UTF8String` literals.
   
   
   ### Why are the changes needed?
   
   To fix `Unsupported literal type` for the following use case:
   
   ```scala
   val min = spark.sql("select min(concat('a', id)) from range(10)").queryExecution.executedPlan.executeCollect().head
   assert(Literal(min.get(0, StringType)) === Literal("a0"))
   ```
   
   ```
   Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a0
   java.lang.RuntimeException: Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a0
   	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:89)
   ```
   
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   
   ### How was this patch tested?
   Unit test.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647251277


   Seems like that's the codes you wrote .. not in the OSS Spark site. You should convert it to the regular Scala types such as `CatalystTypeConverters.createToScalaConverter`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum closed pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #28883:
URL: https://github.com/apache/spark/pull/28883


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647241272


   Yeah, it's internal. Let's don't do this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647125739






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647089277


   **[Test build #124332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124332/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647089393






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647158862






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647089393






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647114256


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124332/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647114252


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647114252






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647125615


   **[Test build #124338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124338/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum removed a comment on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
wangyum removed a comment on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647124877


   > Isn't `UTF8String ` internal type and should be always converted to String while exposing to users?
   
   Yes. I think so.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647124917


   retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HeartSaVioR commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647124876


   Yeah looks like UTF8String is not an end-user facing class, as class doc of UTF8String describes.
   
   https://github.com/apache/spark/blob/d2a656c81ef784657a02e7347bfe87e4331fd2c9/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L42-L51


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647158614


   **[Test build #124338 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124338/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647158862






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647247953


   Actually, I use it internal:
   https://github.com/wangyum/spark/blob/SPARK-27227/sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala#L227-L228


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647124877


   > Isn't `UTF8String ` internal type and should be always converted to String while exposing to users?
   
   Yes. I think so.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647114112


   **[Test build #124332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124332/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647125615


   **[Test build #124338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124338/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647089277


   **[Test build #124332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124332/testReport)** for PR 28883 at commit [`694784b`](https://github.com/apache/spark/commit/694784b336d5601c127fdf52d4eb31ec0ca0daa7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #28883: [SPARK-32042][SQL] Support UTF8String literals

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28883:
URL: https://github.com/apache/spark/pull/28883#issuecomment-647125739






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org