You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by smungee <gi...@git.apache.org> on 2016/05/13 22:45:25 UTC
[GitHub] spark pull request: Fix MapObjects.itemAccessorMethod to handle Ti...
GitHub user smungee opened a pull request:
https://github.com/apache/spark/pull/13108
Fix MapObjects.itemAccessorMethod to handle TimestampType
## What changes were proposed in this pull request?
Fix `MapObjects.itemAccessorMethod` to handle `TimestampType`. Without this fix, this method threw the following error for UDFs that returned `TimestampType`. (Also recommend that this be picked up for the 2.0 release)
` Caused by: scala.MatchError: TimestampType (of class org.apache.spark.sql.types.TimestampType$)
at org.apache.spark.sql.catalyst.expressions.MapObjects.itemAccessorMethod(objects.scala:379)
at org.apache.spark.sql.catalyst.expressions.MapObjects.x$20$lzycompute(objects.scala:414)
at org.apache.spark.sql.catalyst.expressions.MapObjects.x$20(objects.scala:399)
at org.apache.spark.sql.catalyst.expressions.MapObjects.primitiveElement$lzycompute(objects.scala:399)
at org.apache.spark.sql.catalyst.expressions.MapObjects.primitiveElement(objects.scala:399)
at org.apache.spark.sql.catalyst.expressions.MapObjects.doGenCode(objects.scala:450)
at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:105)
at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:102)
at scala.Option.getOrElse(Option.scala:121)`
## How was this patch tested?
Existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/smungee/spark fix-itemAccessorMethod
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13108
----
commit a4b8daf124fc0caf821dc7b2b377167ed3aeb211
Author: Sumedh Mungee <sm...@gmail.com>
Date: 2016-05-13T22:39:41Z
Fix MapObjects.itemAccessorMethod to handle TimestampType
Without this fix, this method throws a MatchError for TimestampType
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix bug where Array[Timestamp] cannot be encod...
Posted by smungee <gi...@git.apache.org>.
Github user smungee commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219198990
@techaddict Added a test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-220497073
@smungee sorry to have this gone stale. Do you mind bringing it up to date? This seems useful to fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix MapObjects.itemAccessorMethod to handle Ti...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219177160
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix MapObjects.itemAccessorMethod to handle Ti...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219191307
(I think it might needs a JIRA because it seems changing existing behaviour)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix MapObjects.itemAccessorMethod to handle Ti...
Posted by smungee <gi...@git.apache.org>.
Github user smungee commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219192693
@HyukjinKwon I think the original author might have just forgotten to add the TimestampType to the match..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-220511906
**[Test build #2999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2999/consoleFull)** for PR 13108 at commit [`387e6c9`](https://github.com/apache/spark/commit/387e6c912191bed1d4d4e09ede92f6ea1cc85a51).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by smungee <gi...@git.apache.org>.
Github user smungee commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-220507839
@rxin The code in question seems to have been removed in this commit by @cloud-fan
https://github.com/apache/spark/commit/c36ca651f9177f8e7a3f6a0098cba5a810ee9deb
With the latest SNAPSHOT version, I no longer see the bug. I've updated this PR to include just my additional test-case, just to make sure it doesn't happen again. I'd recommend adding this test-case in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix bug where Array[Timestamp] cannot be encod...
Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219198790
@smungee Can you please add tests for this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-220514860
LGTM, merging to master and 2.0, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13108
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fix MapObjects.itemAccessorMethod to handle Ti...
Posted by smungee <gi...@git.apache.org>.
Github user smungee commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-219198553
The bug is quite easy to reproduce. In `ExpressionEncoderSuite`, if you add the following test case:
` encodeDecodeTest(Array(Timestamp.valueOf("2016-01-29 10:00:00")), "array of timestamp")
`
... you will see that (without this fix) it fails with the following output:
```
- encode/decode for array of timestamp: [Ljava.sql.Timestamp;@fd9ebde *** FAILED ***
Exception thrown while decoding
Converted: [0,1000000010,800000001,52a7ccdc36800]
Schema: value#61615
root
-- value: array (nullable = true)
|-- element: timestamp (containsNull = true)
Encoder:
class[value[0]: array<timestamp>] (ExpressionEncoderSuite.scala:312)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15321] Fix bug where Array[Timestamp] c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13108#issuecomment-220497014
**[Test build #2999 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2999/consoleFull)** for PR 13108 at commit [`387e6c9`](https://github.com/apache/spark/commit/387e6c912191bed1d4d4e09ede92f6ea1cc85a51).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org