You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gczsjdy <gi...@git.apache.org> on 2017/01/12 08:39:54 UTC
[GitHub] spark pull request #16559: Add expression index and test cases
GitHub user gczsjdy opened a pull request:
https://github.com/apache/spark/pull/16559
Add expression index and test cases
## What changes were proposed in this pull request?
Expression Index returns the value of index n (right) of the array/map (left).
To be consistent with Hive, the left parameter should be of ArrayType or MapType, while the right parameter should be of AtomicType or NullType.
Except the rules above, Hive won't throw any exceptions, even if there's a type mismatch, it will output NULL to indicate not found. This implementation follows the rule.
There's one point this PR temporarily ignored, which is implicit cast for the right parameter, for example, in Hive:
> SELECT index(array(1, 2, 3), '2'); 3
While in this implementation, it returns NULL
## How was this patch tested?
Test cases are also included in this PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gczsjdy/spark udfindex
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16559.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16559
----
commit f98dbb77734fcbd75c8138344ccc357a232ac9bc
Author: GuoChenzhao <ch...@intel.com>
Date: 2017-01-12T08:06:31Z
Add expression index and test cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: [WIP] Add expression index and test cases
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16559
Thanks for your inform @rxin @aray @cloud-fan , I will close this PR.
Sorry for the late reply.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: [WIP] Add expression index and test cases
Posted by aray <gi...@git.apache.org>.
Github user aray commented on the issue:
https://github.com/apache/spark/pull/16559
It can already be done with the `posexplode` UDTF like
```
with t as (values (array(1,2,3)), (array(4,5,6)) as (a))
select col from t lateral view posexplode(a) tt where pos = 2
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: Add expression index and test cases
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy commented on the issue:
https://github.com/apache/spark/pull/16559
cc @chenghao-intel @adrian-wang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: [WIP] Add expression index and test cases
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16559
we already have `GetArrayItem` and `GetMapValue`, and we have special parser rules to support it, e.g. `SELECT array_col[3], map_co['key']`. We can just treat `index` as an alias if `UnresolvedExtractValue`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16559: [WIP] Add expression index and test cases
Posted by gczsjdy <gi...@git.apache.org>.
Github user gczsjdy closed the pull request at:
https://github.com/apache/spark/pull/16559
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: Add expression index and test cases
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16559
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16559: [WIP] Add expression index and test cases
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16559
Do we not have something similar already? cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org