You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/06/10 08:13:07 UTC
[jira] Created: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Add GenericUDF to create arrays, maps, and structs
--------------------------------------------------
Key: HIVE-554
URL: https://issues.apache.org/jira/browse/HIVE-554
Project: Hadoop Hive
Issue Type: New Feature
Reporter: Zheng Shao
Here is an example:
{code}
SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
FROM table
GROUP BY user_id;
{code}
This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772815#action_12772815 ]
Zheng Shao commented on HIVE-554:
---------------------------------
Sorry about the delay in reviewing.
* Can we reuse the same array/map in evaluate() functions? That will improve the efficiency.
* For the display string, can we use "array(1,2,3)" format? Otherwise "[2]" will be ambiguous. Maybe for map, we can also use "map(1:2, 'a':'b')".
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-554:
---------------------------
Attachment: HIVE-554.2.patch
Sorry, I had misinterpreted what you mean by display string.
* Changed getDisplayString() methods to show 'array(...)' and 'map(...)'
* Reused return containers for array and map
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch, HIVE-554.2.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772857#action_12772857 ]
Zheng Shao commented on HIVE-554:
---------------------------------
I mean the functions like this. We can build the string in a little different way. (It applies to both array and map).
I don't think we need JSON format in the explain result, since the expression in explain result is not in JSON anyway.
{code}
+ public String getDisplayString(String[] children) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("[");
+ for(int i=0; i<children.length; i++) {
+ sb.append(children[i]);
+ if(i+1 != children.length) {
+ sb.append(",");
+ }
+ }
+ sb.append("]");
+ return sb.toString();
+ }
+}
+ public String getDisplayString(String[] children) {
+ StringBuilder sb = new StringBuilder();
+ sb.append("[");
+ for(int i=0; i<children.length; i++) {
+ sb.append(children[i]);
+ if(i+1 != children.length) {
+ sb.append(",");
+ }
+ }
+ sb.append("]");
+ return sb.toString();
+ }
+}
{code}
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762340#action_12762340 ]
Zheng Shao commented on HIVE-554:
---------------------------------
array and map are simpler - we can work on that first before getting into struct.
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang reassigned HIVE-554:
------------------------------
Assignee: Paul Yang
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Yang updated HIVE-554:
---------------------------
Attachment: HIVE-554.1.patch
* Added UDF's array() and map() along with respective tests
* When constructing empty arrays and maps, a string Object Inspector is used in initialization.
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-554) Add GenericUDF to create arrays, maps
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-554.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Release Note: HIVE-554. Add GenericUDF to create arrays, maps. (Paul Yang via zshao)
Hadoop Flags: [Reviewed]
Committed. Thanks Paul!
> Add GenericUDF to create arrays, maps
> -------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Fix For: 0.5.0
>
> Attachments: HIVE-554.1.patch, HIVE-554.2.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-554) Add GenericUDF to create arrays, maps
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-554:
----------------------------
Summary: Add GenericUDF to create arrays, maps (was: Add GenericUDF to create arrays, maps, and structs)
struct will be handled later since it requires some changes to the framework (to provide field names).
> Add GenericUDF to create arrays, maps
> -------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch, HIVE-554.2.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-554) Add GenericUDF to create arrays, maps
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772937#action_12772937 ]
Zheng Shao commented on HIVE-554:
---------------------------------
+1. Will commit if tests pass.
> Add GenericUDF to create arrays, maps
> -------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch, HIVE-554.2.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-554) Add GenericUDF to create arrays, maps,
and structs
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772853#action_12772853 ]
Paul Yang commented on HIVE-554:
--------------------------------
It seems that the SerDe's use SerDeUtils.buildJSONString() to get the display string. I could change it there, but then the output would no longer be in JSON. If there were anything that depended on the JSON format, this change would break it as well.
Perhaps we should keep it the way it is?
> Add GenericUDF to create arrays, maps, and structs
> --------------------------------------------------
>
> Key: HIVE-554
> URL: https://issues.apache.org/jira/browse/HIVE-554
> Project: Hadoop Hive
> Issue Type: New Feature
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-554.1.patch
>
>
> Here is an example:
> {code}
> SELECT array(1,2,3)[3], map("a":1,"b":2,"c":3)["a"], struct(user_id:3, revenue: sum(rev))
> FROM table
> GROUP BY user_id;
> {code}
> This is relatively easy to do with the GenericUDF framework, and will greatly increase the flexibility of the language.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.