You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/07 08:28:54 UTC
[GitHub] [spark] Ngone51 opened a new pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Ngone51 opened a new pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR proposes to throw exception by default when user use untyped UDF(a.k.a `org.apache.spark.sql.functions.udf(AnyRef, DataType)`).
And user could still use it by setting `spark.sql.legacy.useUnTypedUdf.enabled` to `true`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
According to #23498, since Spark 3.0, the untyped UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return 0 in Spark 3.0 but null in Spark 2.4. And the behavior change is introduced due to Spark3.0 is built with Scala 2.12 by default.
As a result, this might change data silently and may cause correctness issue if user still expect `null` in some cases. Thus, we'd better to encourage user to use typed UDF to avoid this problem.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
Yeah. User will hit exception now when use untyped UDF.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added test and updated some tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584660789
**[Test build #118246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118246/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074953
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588767302
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23459/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588767302
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23459/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588966699
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584745632
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118246/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496902
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118212/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584649268
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584648510
**[Test build #118243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118243/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584512251
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118216/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583355726
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303653
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22793/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-586963392
LGTM. Also ping @srowen @zhengruifeng Could you help double check ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584512239
**[Test build #118216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118216/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376930319
##########
File path: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
##########
@@ -283,17 +283,20 @@ class FPGrowthModel private[ml] (
.rdd.map(r => (r.getSeq(0), r.getSeq(1)))
.collect().asInstanceOf[Array[(Seq[Any], Seq[Any])]]
val brRules = dataset.sparkSession.sparkContext.broadcast(rules)
+ val dt = associationRules.schema("antecedent").dataType
- val dt = dataset.schema($(itemsCol)).dataType
Review comment:
why change it to `associationRules.schema("antecedent").dataType`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584660789
**[Test build #118246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118246/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545026
**[Test build #118227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118227/testReport)** for PR 27488 at commit [`3297976`](https://github.com/apache/spark/commit/329797634c872247e9ae2e61d2acf6b3465ba9e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106889
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23045/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074392
**[Test build #118721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118721/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589061331
**[Test build #118717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118717/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584649276
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23003/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496101
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22972/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585169886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118287/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510556
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496885
**[Test build #118212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118212/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303644
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585081330
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27488:
[SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #27488: [SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r390032075
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4732,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped Scala UDF, which does not have the input type " +
+ "information. Spark may blindly pass null to the Scala closure with primitive-type " +
+ "argument, and the closure will see the default value of the Java type for the null " +
+ "argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. " +
+ "You could use other typed Scala UDF APIs to avoid this problem, or set " +
Review comment:
In the error message, we should give an example to show how to use the typed Scala UDF for implementing "udf((x: Int) => x, IntegerType)"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589192751
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23478/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584001696
**[Test build #118136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118136/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589192711
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376931550
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4733,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped udf, which does not have the input type information. " +
+ "So, Spark may blindly pass null to the Scala closure with primitive-type argument, " +
+ "and the closure will see the default value of the Java type for the null argument, " +
+ "e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. You could use " +
+ "other typed udf APIs to avoid this problem, or set " +
+ s"${SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF.key} to true to insistently use this."
+ throw new SparkException(errorMsg)
Review comment:
`AnalysisException`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584016278
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376306898
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4733,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_USE_UNTYPED_UDF)) {
+ val errorMsg = "You're using untyped udf, which does not have the input type information. " +
+ "So, Spark may blindly pass null to the Scala closure with primitive-type argument, " +
+ "and the closure will see the default value of the Java type for the null argument, " +
+ "e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. You could use " +
+ "other typed udf APIs to avoid this problem, or set " +
+ "spark.sql.legacy.useUnTypedUdf.enabled to true to insistently use this."
Review comment:
let's not hardcode config names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584661449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23005/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287213
ping @cloud-fan @mengxr @WeichenXu123 please help review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585081341
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118280/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510556
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584652608
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545026
**[Test build #118227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118227/testReport)** for PR 27488 at commit [`3297976`](https://github.com/apache/spark/commit/329797634c872247e9ae2e61d2acf6b3465ba9e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458694
**[Test build #118198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118198/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584004270
**[Test build #118136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118136/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106357
**[Test build #118287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118287/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303653
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22793/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589182114
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118721/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583998835
**[Test build #118127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118127/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588767271
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074953
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588979148
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996127
**[Test build #118127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118127/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584002458
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22900/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584745617
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458694
**[Test build #118198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118198/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996127
**[Test build #118127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118127/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589525950
thanks all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589066978
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589303703
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584002453
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r377427425
##########
File path: mllib/src/main/scala/org/apache/spark/ml/fpm/FPGrowth.scala
##########
@@ -283,17 +283,20 @@ class FPGrowthModel private[ml] (
.rdd.map(r => (r.getSeq(0), r.getSeq(1)))
.collect().asInstanceOf[Array[(Seq[Any], Seq[Any])]]
val brRules = dataset.sparkSession.sparkContext.broadcast(rules)
+ val dt = associationRules.schema("antecedent").dataType
- val dt = dataset.schema($(itemsCol)).dataType
Review comment:
Oh...yeah..we could actually use the original one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584658205
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585169878
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584004277
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r380230473
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
This is a developer API, so I'm wondering if third-party implementations would use primitive type and hit the silent result changing.
I think it's better to ask users to re-compile their Spark application than telling them that they may hit result changinng.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589062620
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057476
**[Test build #118280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118280/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589189245
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584512248
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585509596
Kindly ping @cloud-fan @mengxr @WeichenXu123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496902
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118212/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287388
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583355730
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118028/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584078736
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584015821
**[Test build #118145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118145/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583998854
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118127/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545613
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583289812
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588966708
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118708/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588978589
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588965771
**[Test build #118708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118708/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589303712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118727/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584002458
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22900/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584652628
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118243/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22976/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996786
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22892/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584460898
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589192751
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23478/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584004283
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118136/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589522387
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585056470
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583998848
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106889
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23045/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287031
**[Test build #118024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118024/testReport)** for PR 27488 at commit [`809703e`](https://github.com/apache/spark/commit/809703e77f042b96621b411a596c348c067335ac).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584512251
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118216/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376930736
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4733,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped udf, which does not have the input type information. " +
Review comment:
`untyped Scala UDF`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057784
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584509861
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589303703
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496096
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545626
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22987/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584004277
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584495806
**[Test build #118212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118212/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585081128
**[Test build #118280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118280/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584460902
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118198/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588966708
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118708/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996780
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584460892
**[Test build #118198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118198/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589062620
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584604974
**[Test build #118227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118227/testReport)** for PR 27488 at commit [`3297976`](https://github.com/apache/spark/commit/329797634c872247e9ae2e61d2acf6b3465ba9e7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074392
**[Test build #118721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118721/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584016288
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22907/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583289786
**[Test build #118024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118024/testReport)** for PR 27488 at commit [`809703e`](https://github.com/apache/spark/commit/809703e77f042b96621b411a596c348c067335ac).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545613
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376306049
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
This is a breaking change, but I think it's better than silent result changing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583998848
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588966699
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584744814
**[Test build #118246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118246/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589190667
**[Test build #118727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118727/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376306495
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2006,6 +2006,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_USE_UNTYPED_UDF =
+ buildConf("spark.sql.legacy.useUnTypedUdf.enabled")
Review comment:
I think `spark.sql.legacy.allowUntypedScalaUDF` is better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589182103
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584649276
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23003/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584512248
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588979148
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584014854
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589181640
**[Test build #118721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118721/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589303712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118727/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584661439
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27488:
[SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27488: [SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r390051629
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4732,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped Scala UDF, which does not have the input type " +
+ "information. Spark may blindly pass null to the Scala closure with primitive-type " +
+ "argument, and the closure will see the default value of the Java type for the null " +
+ "argument, e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. " +
+ "You could use other typed Scala UDF APIs to avoid this problem, or set " +
Review comment:
I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496893
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585169121
**[Test build #118287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118287/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r380223797
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
I don't disagree, but this is trading a possible error for a definite error. In light of the recent conversations about not-breaking things, is this wise? (I don't object though.)
Yes, let's restrict this to primitive types. I think Spark ML even uses some UDFs that accept AnyRef or something to work with tuples or triples, IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584661439
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583289817
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118024/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584494961
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584649268
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106879
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588981406
**[Test build #118717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118717/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584460898
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584078748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118145/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r380500081
##########
File path: docs/sql-migration-guide.md
##########
@@ -63,8 +63,8 @@ license: |
- Since Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Set JSON option `inferTimestamp` to `false` to disable such type inferring.
- - In Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(Any, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. Since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
-
+ - Since Spark 3.0, using `org.apache.spark.sql.functions.udf(AnyRef, DataType)` is not allowed by default. Set `spark.sql.legacy.allowUntypedScalaUDF` to true to keep using it. But please note that, in Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(AnyRef, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. However, since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
+
Review comment:
nit: remove spaces here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376305764
##########
File path: docs/sql-migration-guide.md
##########
@@ -65,6 +65,8 @@ license: |
- In Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(Any, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. Since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
+ - Since Spark 3.0, using `org.apache.spark.sql.functions.udf(AnyRef, DataType)` is not allowed by default. Set `spark.sql.legacy.useUnTypedUdf.enabled` to true to keep use it.
Review comment:
can we merge the migration guide between this one and the one that changes the behavior?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584460902
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118198/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376931317
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4733,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped udf, which does not have the input type information. " +
+ "So, Spark may blindly pass null to the Scala closure with primitive-type argument, " +
+ "and the closure will see the default value of the Java type for the null argument, " +
+ "e.g. `udf((x: Int) => x, IntegerType)`, the result is 0 for null input. You could use " +
+ "other typed udf APIs to avoid this problem, or set " +
+ s"${SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF.key} to true to insistently use this."
Review comment:
`to true and use this API with caution`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589182114
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118721/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584605360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118227/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588979159
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23468/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287031
**[Test build #118024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118024/testReport)** for PR 27488 at commit [`809703e`](https://github.com/apache/spark/commit/809703e77f042b96621b411a596c348c067335ac).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376930929
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##########
@@ -4732,6 +4733,15 @@ object functions {
* @since 2.0.0
*/
def udf(f: AnyRef, dataType: DataType): UserDefinedFunction = {
+ if (!SQLConf.get.getConf(SQLConf.LEGACY_ALLOW_UNTYPED_SCALA_UDF)) {
+ val errorMsg = "You're using untyped udf, which does not have the input type information. " +
+ "So, Spark may blindly pass null to the Scala closure with primitive-type argument, " +
Review comment:
`So, Spark ...` -> `Spark ...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583289817
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118024/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584004283
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118136/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057793
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23039/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584745632
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118246/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287393
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22789/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585169886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118287/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584078736
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588765729
**[Test build #118708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118708/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584077883
**[Test build #118145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118145/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584001696
**[Test build #118136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118136/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584016278
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458922
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584745617
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589062658
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118717/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584605355
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057793
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23039/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r380143742
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
We can avoid this breaking change if we know that the type parameter won't be primitive types. cc @srowen @zhengruifeng
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287393
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22789/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589192711
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584015821
**[Test build #118145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118145/testReport)** for PR 27488 at commit [`93bf652`](https://github.com/apache/spark/commit/93bf65263dcde4aab2c6d0232bde74ab30030fec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584652608
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583289812
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584016288
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22907/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996780
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589190667
**[Test build #118727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118727/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376929615
##########
File path: docs/sql-migration-guide.md
##########
@@ -63,8 +63,8 @@ license: |
- Since Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Set JSON option `inferTimestamp` to `false` to disable such type inferring.
- - In Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(Any, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. Since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
-
+ - Since Spark 3.0, using `org.apache.spark.sql.functions.udf(AnyRef, DataType)` is not allowed by default. Set `spark.sql.legacy.allowUntypedScalaUDF` to true to keep use it. But please note that, in Spark version 2.4 and earlier, if `org.apache.spark.sql.functions.udf(AnyRef, DataType)` gets a Scala closure with primitive-type argument, the returned UDF will return null if the input values is null. However, since Spark 3.0, the UDF will return the default value of the Java type if the input value is null. For example, `val f = udf((x: Int) => x, IntegerType)`, `f($"x")` will return null in Spark 2.4 and earlier if column `x` is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.
Review comment:
`to keep use it` -> `to keep using it`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588767271
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584652628
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118243/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458926
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22960/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583287388
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458926
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22960/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303644
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584605355
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589062658
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118717/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23472/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r376267347
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
`TypeTag` is required for typed UDF when create udf for `createTransformFunc`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584661449
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23005/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22976/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496101
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22972/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496893
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510148
**[Test build #118216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118216/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584652576
**[Test build #118243 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118243/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
* This patch **fails build dependency tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584648510
**[Test build #118243 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118243/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303141
**[Test build #118028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118028/testReport)** for PR 27488 at commit [`23e52ea`](https://github.com/apache/spark/commit/23e52eaaac443dc3c68ed82ef5e2b4181e8e8ea5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583998854
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118127/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583999361
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584495806
**[Test build #118212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118212/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583355726
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057784
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588979159
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23468/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585081330
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583303141
**[Test build #118028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118028/testReport)** for PR 27488 at commit [`23e52ea`](https://github.com/apache/spark/commit/23e52eaaac443dc3c68ed82ef5e2b4181e8e8ea5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583996786
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22892/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585103541
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584458922
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584510148
**[Test build #118216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118216/testReport)** for PR 27488 at commit [`549cdf4`](https://github.com/apache/spark/commit/549cdf4d1d22f34091439cc47f486dd76e2a0de8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589074966
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23472/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585057476
**[Test build #118280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118280/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583355387
**[Test build #118028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118028/testReport)** for PR 27488 at commit [`23e52ea`](https://github.com/apache/spark/commit/23e52eaaac443dc3c68ed82ef5e2b4181e8e8ea5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584545626
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22987/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588765729
**[Test build #118708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118708/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106879
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-588981406
**[Test build #118717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118717/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589302753
**[Test build #118727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118727/testReport)** for PR 27488 at commit [`0d1601b`](https://github.com/apache/spark/commit/0d1601bde2fc4ab39ed6b3c82e486e9c1174b915).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585169878
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585106357
**[Test build #118287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118287/testReport)** for PR 27488 at commit [`6969596`](https://github.com/apache/spark/commit/6969596ce7d1ce1b2962079629a1701345f349ec).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-589182103
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r380230473
##########
File path: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala
##########
@@ -79,7 +80,7 @@ abstract class Transformer extends PipelineStage {
* result as a new column.
*/
@DeveloperApi
-abstract class UnaryTransformer[IN, OUT, T <: UnaryTransformer[IN, OUT, T]]
+abstract class UnaryTransformer[IN: TypeTag, OUT: TypeTag, T <: UnaryTransformer[IN, OUT, T]]
Review comment:
This is a developer API, so I'm wondering if third-party implementations would use primitive type and hit the silent result changing.
I think it's better to ask users to re-compile their Spark application than just telling them that they may hit result changinng.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584002453
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584496096
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584605360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118227/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-584078748
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118145/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-583355730
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118028/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27488:
[SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #27488: [SPARK-31010][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#discussion_r390031663
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2016,6 +2016,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val LEGACY_ALLOW_UNTYPED_SCALA_UDF =
+ buildConf("spark.sql.legacy.allowUntypedScalaUDF")
+ .internal()
+ .doc("When set to true, user is allowed to use org.apache.spark.sql.functions." +
+ "udf(f: AnyRef, dataType: DataType). Otherwise, exception will be throw.")
Review comment:
> exception will be throw
->
> an exception will be thrown at runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27488:
[SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by
default
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27488: [SPARK-26580][SQL][ML][FOLLOW-UP] Throw exception when use untyped UDF by default
URL: https://github.com/apache/spark/pull/27488#issuecomment-585081341
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118280/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org