You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by priyankagargnitk <gi...@git.apache.org> on 2018/06/30 07:29:45 UTC
[GitHub] spark pull request #21679: SPARK-24695: To add support to return Calendar in...
GitHub user priyankagargnitk opened a pull request:
https://github.com/apache/spark/pull/21679
SPARK-24695: To add support to return Calendar interval from udf.
## What changes were proposed in this pull request?
This change adds capability to return Calender interval from udf.
Earlier, the udf of Type (String => CalendarInterval) was throwing Exception stating:
Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
java.lang.UnsupportedOperationException: Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
at org.apache.spark.sql.catalyst.ScalaReflection391anonfun.apply(ScalaReflection.scala:781)
## How was this patch tested?
Added test case in ScalaReflectionSuite.scala and ExpressionEncoderSuite.scala
Also, tested by creating an udf that returns Calendar interval.
jira entry for detail: https://issues.apache.org/jira/browse/SPARK-24695
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/priyankagargnitk/spark SPARK-24695
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21679.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21679
----
commit bd805299cc9802597c165a6de1667a7b02ad48ae
Author: Priyanka Garg <pr...@...>
Date: 2018-06-30T07:23:56Z
SPARK-24695: To add support to return Calender interval from udf.
## What changes were proposed in this pull request?
This change adds capability to return Calender interval from udf.
Earlier, the udf of Type (String => CalendarInterval) was throwing Exception stating:
Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
java.lang.UnsupportedOperationException: Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
at org.apache.spark.sql.catalyst.ScalaReflection391anonfun.apply(ScalaReflection.scala:781)
## How was this patch tested?
Added test case in ScalaReflectionSuite.scala and ExpressionEncoderSuite.scala
Also, tested by creating an udf that returns Calendar interval.
jira entry for detail: https://issues.apache.org/jira/browse/SPARK-24695
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21679
For your information, @cloud-fan and I are discussing how we expose calendar interval data types. Will post the proposal later.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: SPARK-24695: To add support to return Calendar interval ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21679
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: SPARK-24695: To add support to return Calendar interval ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21679
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21679
I would better open a discussion thread in the mailing list before making a PR to expose it. Shell we leave this close this meanwhile? I think it's better to leave active and actionable PRs only.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21679: [SPARK-24695] [SQL]: To add support to return Cal...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21679
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21679
I think we should close this for now then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21679
Yea, so it's kind of half public now but not completely exposed. It should be exposed before we extend its support.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21679
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21679
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by priyankagargnitk <gi...@git.apache.org>.
Github user priyankagargnitk commented on the issue:
https://github.com/apache/spark/pull/21679
org.apache.spark.unsafe.types.CalenderInterval is already public, am i missing something.
Also, what if i want to do some computation on any data type and return Calender Interval.. How should i solve this problem in current scenario.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: SPARK-24695: To add support to return Calendar interval ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21679
btw, can you update the title like `[SPARK-24695][SQL]...`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: SPARK-24695: To add support to return Calendar interval ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21679
Since `CalendarInterval` is an internal class, I think users are not intended to use the class directly...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21679
Actually, the unsafe package does not include user-facing classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: SPARK-24695: To add support to return Calendar interval ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21679
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21679: [SPARK-24695] [SQL]: To add support to return Calendar i...
Posted by priyankagargnitk <gi...@git.apache.org>.
Github user priyankagargnitk commented on the issue:
https://github.com/apache/spark/pull/21679
What if i make changes to expose it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org