You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "John Sichi (JIRA)" <ji...@apache.org> on 2010/04/08 22:54:36 UTC
[jira] Created: (HIVE-1295) facilitate HBase bulk loads from Hive
facilitate HBase bulk loads from Hive
-------------------------------------
Key: HIVE-1295
URL: https://issues.apache.org/jira/browse/HIVE-1295
Project: Hadoop Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.6.0
Reporter: John Sichi
Assignee: John Sichi
Fix For: 0.6.0
HBase supports a bulk load procedure:
http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856186#action_12856186 ]
John Sichi commented on HIVE-1295:
----------------------------------
The copyright grant issue is INFRA-2605.
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1295:
-----------------------------
Attachment: HIVE-1295.1.patch
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858018#action_12858018 ]
John Sichi commented on HIVE-1295:
----------------------------------
While testing on a different data set, I found some bugs, so I'll submit a followup patch for that.
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860064#action_12860064 ]
John Sichi commented on HIVE-1295:
----------------------------------
Followup logged as HIVE-1321.
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856173#action_12856173 ]
John Sichi commented on HIVE-1295:
----------------------------------
I'm still working on testing out the procedure beyond toy cases; here's the doc:
http://wiki.apache.org/hadoop/Hive/HBaseBulkLoad
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856714#action_12856714 ]
Namit Jain commented on HIVE-1295:
----------------------------------
I will take a look
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1295:
-----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed. Thanks John
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1295) facilitate HBase bulk loads from Hive
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1295:
-----------------------------
Status: Patch Available (was: Open)
Notes on the patch:
* After the recent Apache JIRA downtime, the ASF copyright license assignment radio button is no longer here in the UI, so I couldn't click that to grant to Apache, but I hereby grant it etc etc (and I'm a Facebook employee anyway, so I think my contribution is covered under the corporate license assignment).
* The unit test requires MiniMR mode (instead of local mode) because it needs multiple reducers in order for the custom partitioner to kick in. Accordingly, I named the file with a .m suffix (instead of .q) to keep it segregated from the other unit tests. To run it, use -Dtestcase=TestHBaseMinimrCliDriver.
* I had to make one change to the classpath definition in build-common.xml to avoid a webapp ordering issue from HBase and Hadoop being loaded in the same Jetty instance.
* I made one change to ql/build.xml's classpath (adding jsp-2.1 jars). This was not actually needed for my hbase-handler test to run, but it fixes an existing breakage with running ql tests with -Dclustermode=miniMR, needed for HIVE-117. (The job tracker wasn't starting because it wasn't able to load some JSP API classes.)
*
> facilitate HBase bulk loads from Hive
> -------------------------------------
>
> Key: HIVE-1295
> URL: https://issues.apache.org/jira/browse/HIVE-1295
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: HBase Handler
> Affects Versions: 0.6.0
> Reporter: John Sichi
> Assignee: John Sichi
> Fix For: 0.6.0
>
> Attachments: HIVE-1295.1.patch
>
>
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk load files to be prepared inside of Hive via a sequence of SQL statements and then pushed into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a single column family.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira