You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/11/17 01:01:13 UTC
[jira] Created: (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Consolidate and document completebulkload tool and loadtable.rb
---------------------------------------------------------------
Key: HBASE-3240
URL: https://issues.apache.org/jira/browse/HBASE-3240
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HBASE-3240:
----------------------------------
Attachment: hbase-3240.1.patch
Updated patch to address Todd's comment.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3240:
-------------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk, thanks atm.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051191#comment-13051191 ]
Aaron T. Myers commented on HBASE-3240:
---------------------------------------
Hey Doug, any update on these docs? If you don't have time to get to these, do you mind if I reassign the issue?
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Doug Meil
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HBASE-3240:
----------------------------------
Status: Patch Available (was: Open)
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060208#comment-13060208 ]
Todd Lipcon commented on HBASE-3240:
------------------------------------
one more thing: you've edited the usage information in the docs, but not in ImportTsv.java itself. The usage info printed on the command line should match what's shown in the docs.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060167#comment-13060167 ]
Todd Lipcon commented on HBASE-3240:
------------------------------------
+ this data into HBase. This tool by default takes care of both steps of
+ bulk loading as described above. This tool is available by running
importtsv doesn't complete the bulk load, does it?
- -Dimporttsv.timestamp=currentTimeAsLong - use the specified timestamp for the import
why'd you remove this? it no longer exists?
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-3240:
-----------------------------
Assignee: Doug Meil
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Doug Meil
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932721#action_12932721 ]
Todd Lipcon commented on HBASE-3240:
------------------------------------
There's also some question as to whether loadtable.rb will work at all on the new master, since we no longer have a MetaScanner chore.. if that's the case, this will have to move to be an 0.90 blocker.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HBASE-3240) Improve documentation of importtsv
and bulk loads
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-3240:
-------------------------------
Summary: Improve documentation of importtsv and bulk loads (was: Consolidate and document completebulkload tool and loadtable.rb)
> Improve documentation of importtsv and bulk loads
> -------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Improve documentation of importtsv
and bulk loads
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060221#comment-13060221 ]
Hudson commented on HBASE-3240:
-------------------------------
Integrated in HBase-TRUNK #2006 (See [https://builds.apache.org/job/HBase-TRUNK/2006/])
HBASE-3240 Improve documentation of importtsv and bulk loads.
todd :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
* /hbase/trunk/src/site/xdoc/bulk-loads.xml
> Improve documentation of importtsv and bulk loads
> -------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017280#comment-13017280 ]
stack commented on HBASE-3240:
------------------------------
The loadtable.rb script does this on invocation:
{code}
puts 'DISABLED!!!! Use completebulkload instead. See tail of http://hbase.apache.org/bulk-loads.html'
{code}
To close this issue, docs. need a bit of an update.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990243#comment-12990243 ]
stack commented on HBASE-3240:
------------------------------
loadtable.rb does not work in 0.90.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029341#comment-13029341 ]
Doug Meil commented on HBASE-3240:
----------------------------------
We'll do the docs on this.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Improve documentation of importtsv
and bulk loads
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061095#comment-13061095 ]
Lars George commented on HBASE-3240:
------------------------------------
I think the "-c" is wrong? The GenericOptionsParser only takes "-conf":
{code}
Option oconf = OptionBuilder.withArgName("configuration file")
.hasArg()
.withDescription("specify an application configuration file")
.create("conf");
{code}
Same in trunk https://github.com/apache/hadoop/blob/trunk/src/core/org/apache/hadoop/util/GenericOptionsParser.java#L206
Or is this handled somewhere else?
> Improve documentation of importtsv and bulk loads
> -------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Fix For: 0.92.0
>
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Doug Meil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Meil updated HBASE-3240:
-----------------------------
Assignee: Aaron T. Myers (was: Doug Meil)
Aaron, sorry about this. The guy on our team that was going to do this was swamped, so I re-assigned this to you.
Go for it!
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060172#comment-13060172 ]
Aaron T. Myers commented on HBASE-3240:
---------------------------------------
bq. importtsv doesn't complete the bulk load, does it?
My bad. I misunderstood the implementation of importtsv as always using bulk loads, but optionally not actually doing complete. In fact, it either prepares HFOFs for bulk load, or just uses the HBase put API directly. I'll update the patch to make this more clear.
bq. why'd you remove this? it no longer exists?
It was duplicated two lines down. Note this page: http://hbase.apache.org/bulk-loads.html
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060209#comment-13060209 ]
Todd Lipcon commented on HBASE-3240:
------------------------------------
nm, was looking at the wrong source dir. I see you have done that. Will commit.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch, hbase-3240.1.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3240) Consolidate and document
completebulkload tool and loadtable.rb
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HBASE-3240:
----------------------------------
Attachment: hbase-3240.0.patch
Here's a patch which updates/reorganizes the documentation regarding HBase's bulk load features. Nothing in the previous documentation was in fact incorrect, but this patch makes a few things more explicit.
> Consolidate and document completebulkload tool and loadtable.rb
> ---------------------------------------------------------------
>
> Key: HBASE-3240
> URL: https://issues.apache.org/jira/browse/HBASE-3240
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.92.0
> Reporter: Todd Lipcon
> Assignee: Aaron T. Myers
> Attachments: hbase-3240.0.patch
>
>
> Right now our bulk load features are a little confusing. We have loadtable.rb for new tables and completebulkload for existing tables. The docs only talk about the incremental case, and there are basically no docs for the ruby script. We should conslidate these things and make the documentation a little more clear on the full story.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira