You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/29 11:38:14 UTC
[GitHub] [spark] Peng-Lei opened a new pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Peng-Lei opened a new pull request #33132:
URL: https://github.com/apache/spark/pull/33132
### What changes were proposed in this pull request?
1. The function `width_bucket` is introduced from [SPARK-21117](https://issues.apache.org/jira/browse/SPARK-21117)
2. The YearMonthIntervalType is just store the value with Int.
3. Modify the `inputType` to allow the width_bucket to support the YearMonthIntervalType
### Why are the changes needed?
[35926](https://issues.apache.org/jira/browse/SPARK-35926)
### Does this PR introduce _any_ user-facing change?
Yes. The user can use `width_bucket` with Period type.
### How was this patch tested?
Add ut test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-944241305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-944252722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-944188409
**[Test build #144302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144302/testReport)** for PR 33132 at commit [`e8f914b`](https://github.com/apache/spark/commit/e8f914b23dbe5949afe7c0cb6a134ae26a772b6e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874496061
> > Why are the changes needed?
> > 35926
>
> @Peng-Lei I would like to see why the changes are really needed. Could you explain that, please.
>
> Just in case, I would like to propose to focus on feature parity with `CalendarIntervalType` in the release 3.2. So, users could smoothly replace it by new ANSI types. Could you re-check Spark's code base, and point out the places where we miss handling of the ANSI interval types.
@MaxGekk Firstly, the `WIDTH_BUCKET` function assigns values to buckets (individual segments) in an equiwidth histogram. The ANSI SQL Standard Syntax is like follow: `WIDTH_BUCKET( expression, min, max, buckets)`. [Reference](https://www.oreilly.com/library/view/sql-in-a/9780596155322/re91.html). Secondly, `WIDTH_BUCKET` just support `Double` at now, Of course, we can cast `Int` to `Double` to use it. But we cloud not cast `YearMonthIntervayType` to `Double`. Finally, I think it has a use scenario. eg: Histogram of employee years of service, the `years of service` is a column of `YearMonthIntervalType` dataType.
Maybe I'm wrong. When I studied the `WIDTH_BUCKET` code, I wondered if I could support `YearMonthIntervalType`. Thank you for reminding me. I will check whether there are miss handling of the ANSI interval types and test ANSI interval types features added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874550183
> Firstly, ... Secondly, ... Finally, ...
Thank you for the explanation. Could you add the text to the section `Why are the changes needed?` in PR's description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943218896
**[Test build #144254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144254/testReport)** for PR 33132 at commit [`9096855`](https://github.com/apache/spark/commit/909685559d4a197c6eaef6ab4eb17fbf3e2205ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943458792
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144254/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-870712291
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943448380
**[Test build #144254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144254/testReport)** for PR 33132 at commit [`9096855`](https://github.com/apache/spark/commit/909685559d4a197c6eaef6ab4eb17fbf3e2205ea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-870540399
@MaxGekk Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-944252722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874272032
> Why are the changes needed?
> 35926
@Peng-Lei I would like to see why the changes are really needed. Could you explain that, please.
Just in case, I would like to propose to focus on feature parity with `CalendarIntervalType` in the release 3.2. So, users could smoothly replace it by new ANSI types. Could you re-check Spark's code base, and point out the places where we miss handling of the ANSI interval types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-870712291
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943223813
@MaxGekk @sarutak Could you take a look ? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943458792
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144254/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-870540399
@MaxGekk Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874272032
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei edited a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei edited a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874496061
> > Why are the changes needed?
> > 35926
>
> @Peng-Lei I would like to see why the changes are really needed. Could you explain that, please.
>
> Just in case, I would like to propose to focus on feature parity with `CalendarIntervalType` in the release 3.2. So, users could smoothly replace it by new ANSI types. Could you re-check Spark's code base, and point out the places where we miss handling of the ANSI interval types.
@MaxGekk
Firstly, the `WIDTH_BUCKET` function assigns values to buckets (individual segments) in an equiwidth histogram. The ANSI SQL Standard Syntax is like follow: `WIDTH_BUCKET( expression, min, max, buckets)`. [Reference](https://www.oreilly.com/library/view/sql-in-a/9780596155322/re91.html).
Secondly, `WIDTH_BUCKET` just support `Double` at now, Of course, we can cast `Int` to `Double` to use it. But we cloud not cast `YearMonthIntervayType` to `Double`.
Finally, I think it has a use scenario. eg: Histogram of employee years of service, the `years of service` is a column of `YearMonthIntervalType` dataType.
Maybe I'm wrong. When I studied the `WIDTH_BUCKET` code, I wondered if I could support `YearMonthIntervalType`. Thank you for reminding me. I will check whether there are miss handling of the ANSI interval types and test ANSI interval types features added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874496061
> > Why are the changes needed?
> > 35926
>
> @Peng-Lei I would like to see why the changes are really needed. Could you explain that, please.
>
> Just in case, I would like to propose to focus on feature parity with `CalendarIntervalType` in the release 3.2. So, users could smoothly replace it by new ANSI types. Could you re-check Spark's code base, and point out the places where we miss handling of the ANSI interval types.
@MaxGekk Firstly, the `WIDTH_BUCKET` function assigns values to buckets (individual segments) in an equiwidth histogram. The ANSI SQL Standard Syntax is like follow: `WIDTH_BUCKET( expression, min, max, buckets)`. [Reference](https://www.oreilly.com/library/view/sql-in-a/9780596155322/re91.html). Secondly, `WIDTH_BUCKET` just support `Double` at now, Of course, we can cast `Int` to `Double` to use it. But we cloud not cast `YearMonthIntervayType` to `Double`. Finally, I think it has a use scenario. eg: Histogram of employee years of service, the `years of service` is a column of `YearMonthIntervalType` dataType.
Maybe I'm wrong. When I studied the `WIDTH_BUCKET` code, I wondered if I could support `YearMonthIntervalType`. Thank you for reminding me. I will check whether there are miss handling of the ANSI interval types and test ANSI interval types features added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943254225
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943300484
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943294293
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-944338862
+1, LGTM. Merged to master.
Thank you, @Peng-Lei .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943218896
**[Test build #144254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144254/testReport)** for PR 33132 at commit [`9096855`](https://github.com/apache/spark/commit/909685559d4a197c6eaef6ab4eb17fbf3e2205ea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-943300484
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #33132:
URL: https://github.com/apache/spark/pull/33132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei edited a comment on pull request #33132: [SPARK-35926][SQL] Add support YearMonthIntervalType for width_bucket
Posted by GitBox <gi...@apache.org>.
Peng-Lei edited a comment on pull request #33132:
URL: https://github.com/apache/spark/pull/33132#issuecomment-874496061
> > Why are the changes needed?
> > 35926
>
> @Peng-Lei I would like to see why the changes are really needed. Could you explain that, please.
>
> Just in case, I would like to propose to focus on feature parity with `CalendarIntervalType` in the release 3.2. So, users could smoothly replace it by new ANSI types. Could you re-check Spark's code base, and point out the places where we miss handling of the ANSI interval types.
@MaxGekk
Firstly, the `WIDTH_BUCKET` function assigns values to buckets (individual segments) in an equiwidth histogram. The ANSI SQL Standard Syntax is like follow: `WIDTH_BUCKET( expression, min, max, buckets)`. [Reference](https://www.oreilly.com/library/view/sql-in-a/9780596155322/re91.html).
Secondly, `WIDTH_BUCKET` just support `Double` at now, Of course, we can cast `Int` to `Double` to use it. But we cloud not cast `YearMonthIntervayType` to `Double`.
Finally, I think it has a use scenario. eg: Histogram of employee years of service, the `years of service` is a column of `YearMonthIntervalType` dataType.
Maybe I'm wrong. When I studied the `WIDTH_BUCKET` code, I wondered if I could support `YearMonthIntervalType`. Thank you for reminding me. I will check whether there are miss handling of the ANSI interval types and test ANSI interval types features added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org