You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2008/10/31 17:45:44 UTC
[jira] Created: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
New Backup Export/Import MR for 0.18/0.19
-----------------------------------------
Key: HBASE-974
URL: https://issues.apache.org/jira/browse/HBASE-974
Project: Hadoop HBase
Issue Type: New Feature
Affects Versions: 0.18.1, 0.18.0, 0.19.0
Reporter: Jonathan Gray
Priority: Minor
Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672899#action_12672899 ]
Erik Holstad commented on HBASE-974:
------------------------------------
First I didn't really understand why I created I new HTable in every reducer, but today it
struck me, that we had it setup in another way. We have kinds like a pool of tables that
you check in and out, but it has dependencies so that is why I removed it. Of course it
doesn't make any sense to have it the way it is now, it just slows things down a lot.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-974:
-------------------------------
Attachment: makeImportJar.sh
Removed the dependency to HBaseRef since it is not needed.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-974:
-------------------------------
Attachment: ExportMR.sh
ImportMR.sh
makeExportJar.sh
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-974:
-------------------------------
Attachment: (was: makeImportJar.sh)
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "atppp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733118#action_12733118 ]
atppp commented on HBASE-974:
-----------------------------
very interesting idea of directly serializing RR. However, in importer reducer, as you said, you could create new HTable in configure(), but you don't even have to do that. You can just directly let output collect (key, batchUpdate) and TableReduce would take care of committing. Plus TableReduce sets autoflush off which _significantly_ boosts importing performance.
see original example in HBASE-897 and my recent changes. (should mark 897 duplicate or close it?)
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-974.
---------------------------------
Resolution: Won't Fix
Issue contains tools to perform this on 0.18 and 0.19. No plans to commit any of this into branches.
Closing issue as Won't Fix. 0.20 backup now being worked on in HBASE-1684
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-974:
-----------------------------------
Assignee: Erik Holstad
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671532#action_12671532 ]
Erik Holstad commented on HBASE-974:
------------------------------------
Hey Stack!
I did some changes to the files so that they are looking better according to the standards and are using the
util.Writables instead.
Haven't tested the functionality yet, so will post the new code when I have, so you can comment on it again :)
Not really sure where it fits best, but would say examples for now.
Don't want to spend too much time fixing the code though, since I think that we will have more efficient ways
of doing the backup in a little, when we start messing with Cascading for example, but will see what Yair says
and after that post the updated code.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-974:
-------------------------------
Attachment: makeImportJar.sh
ExportMR.java
ImportMR.java
MapReduce files for importing and exporting data from HBase tables.
Extra scripts used to create the jars and execute the jobs are attached for reference.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ImportMR.java, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671525#action_12671525 ]
stack commented on HBASE-974:
-----------------------------
Erik:
Would suggest that when you are successful with Yair, that you get him to +1 one this issue. On the ExportMR job, I wonder if its possible to set maps == 0 so you don't have to supply a map task? Should we commit these classes to hbase? Into a contrib or under examples? I like the way they serialize RR. Good idea. If we're going to commit, they need apache licenses and the style fixed up (don't ask jgray -- he'll only tell you wrong thing.... smile). For the below, check Writables in hbase util. I think there are methods there to help you do the below:
{code}
ByteArrayInputStream bis = new ByteArrayInputStream( ((BytesWritable)val).get() );// baos.toByteArray());
DataInputStream dis = new DataInputStream(bis);
RowResult rowRes = new RowResult();
rowRes.readFields(dis);
{code}
If you use HbaseMapWritable instead of MW, you could do without Text and toString'ing table name (I think). In 880, I believe RowResult and BatchUpdate have same ancestor. Would be sweet if they could be used interchangeably so you wouldn't need to do the conversion in rowResultToBatchUpdate. You think it makes sense creating the new HTable in the reduce each time its invoked and not in its configure step?
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733259#action_12733259 ]
Jonathan Gray commented on HBASE-974:
-------------------------------------
TableReduce has had some performance issues in the past. I think it's pretty good now though.
Yes, you'll definitely want to turn off autoflush. Honestly I don't think that option existed when these jobs were written :)
I will close these issues once I open an 0.20 issue.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
> Attachments: ExportMR.java, ExportMR.sh, ImportMR.java, ImportMR.sh, makeExportJar.sh, makeImportJar.sh
>
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by stack <st...@duboce.net>.
Adding as subpackage sounds good or should we start up a contrib or tools
section?
Dan's import/export is text-only? Whereas Eriks does binary. Would we
check in both or merge the two?
St.Ack
On Mon, Nov 3, 2008 at 1:40 AM, Dingding Ye <ye...@gmail.com> wrote:
> yeah. Dan's previous work is the patch from Mahout.
>
> I think we can add the tool as a sub-package of
> org.apache.hadoop.hbase.mapred
>
> I need to clear up something. I have no available server for hosting files,
> ;)
>
> On Mon, Nov 3, 2008 at 5:34 PM, Cosmin Lehene <cl...@adobe.com> wrote:
>
> > Hi,
> >
> > We were also working on this tool. Basically we made it work with 0.18
> and
> > added CRC. Could you send a link with the code? Maybe we could merge it.
> > Is there any plan to include this as a HBase tool in the codebase?
> > Currently it seems there's only a patch from Mahout.
> >
> > Thanks,
> > Cosmin
> >
> >
> > On 11/3/08 11:26 AM, "sishen (JIRA)" <ji...@apache.org> wrote:
> >
> >
> >
> > [
> >
> https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644670#action_12644670
> ]
> >
> > sishen commented on HBASE-974:
> > ------------------------------
> >
> > I have patched Dan's previous work on this tool to work with 0.18/0.19.
> >
> > However, looking forward to see this effort.
> >
> > > New Backup Export/Import MR for 0.18/0.19
> > > -----------------------------------------
> > >
> > > Key: HBASE-974
> > > URL: https://issues.apache.org/jira/browse/HBASE-974
> > > Project: Hadoop HBase
> > > Issue Type: New Feature
> > > Affects Versions: 0.18.0, 0.18.1, 0.19.0
> > > Reporter: Jonathan Gray
> > > Assignee: Erik Holstad
> > > Priority: Minor
> > >
> > > Erik has built a new backup tool that's compatible with newer versions
> of
> > HBase.
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
> >
>
Re: [jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by Cosmin Lehene <cl...@adobe.com>.
Great, thanks!
Though, org.apache.hadoop.hbase.mapred looks pretty crowded... :)
Cosmin
On 11/3/08 11:40 AM, "Dingding Ye" <ye...@gmail.com> wrote:
yeah. Dan's previous work is the patch from Mahout.
I think we can add the tool as a sub-package of
org.apache.hadoop.hbase.mapred
I need to clear up something. I have no available server for hosting files,
;)
On Mon, Nov 3, 2008 at 5:34 PM, Cosmin Lehene <cl...@adobe.com> wrote:
> Hi,
>
> We were also working on this tool. Basically we made it work with 0.18 and
> added CRC. Could you send a link with the code? Maybe we could merge it.
> Is there any plan to include this as a HBase tool in the codebase?
> Currently it seems there's only a patch from Mahout.
>
> Thanks,
> Cosmin
>
>
> On 11/3/08 11:26 AM, "sishen (JIRA)" <ji...@apache.org> wrote:
>
>
>
> [
> https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644670#action_12644670]
>
> sishen commented on HBASE-974:
> ------------------------------
>
> I have patched Dan's previous work on this tool to work with 0.18/0.19.
>
> However, looking forward to see this effort.
>
> > New Backup Export/Import MR for 0.18/0.19
> > -----------------------------------------
> >
> > Key: HBASE-974
> > URL: https://issues.apache.org/jira/browse/HBASE-974
> > Project: Hadoop HBase
> > Issue Type: New Feature
> > Affects Versions: 0.18.0, 0.18.1, 0.19.0
> > Reporter: Jonathan Gray
> > Assignee: Erik Holstad
> > Priority: Minor
> >
> > Erik has built a new backup tool that's compatible with newer versions of
> HBase.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
Re: [jira] Commented: (HBASE-974) New Backup Export/Import MR for 0.18/0.19
Posted by Dingding Ye <ye...@gmail.com>.
yeah. Dan's previous work is the patch from Mahout.
I think we can add the tool as a sub-package of
org.apache.hadoop.hbase.mapred
I need to clear up something. I have no available server for hosting files,
;)
On Mon, Nov 3, 2008 at 5:34 PM, Cosmin Lehene <cl...@adobe.com> wrote:
> Hi,
>
> We were also working on this tool. Basically we made it work with 0.18 and
> added CRC. Could you send a link with the code? Maybe we could merge it.
> Is there any plan to include this as a HBase tool in the codebase?
> Currently it seems there's only a patch from Mahout.
>
> Thanks,
> Cosmin
>
>
> On 11/3/08 11:26 AM, "sishen (JIRA)" <ji...@apache.org> wrote:
>
>
>
> [
> https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644670#action_12644670]
>
> sishen commented on HBASE-974:
> ------------------------------
>
> I have patched Dan's previous work on this tool to work with 0.18/0.19.
>
> However, looking forward to see this effort.
>
> > New Backup Export/Import MR for 0.18/0.19
> > -----------------------------------------
> >
> > Key: HBASE-974
> > URL: https://issues.apache.org/jira/browse/HBASE-974
> > Project: Hadoop HBase
> > Issue Type: New Feature
> > Affects Versions: 0.18.0, 0.18.1, 0.19.0
> > Reporter: Jonathan Gray
> > Assignee: Erik Holstad
> > Priority: Minor
> >
> > Erik has built a new backup tool that's compatible with newer versions of
> HBase.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
Re: [jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by Cosmin Lehene <cl...@adobe.com>.
Hi,
We were also working on this tool. Basically we made it work with 0.18 and added CRC. Could you send a link with the code? Maybe we could merge it.
Is there any plan to include this as a HBase tool in the codebase? Currently it seems there's only a patch from Mahout.
Thanks,
Cosmin
On 11/3/08 11:26 AM, "sishen (JIRA)" <ji...@apache.org> wrote:
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644670#action_12644670 ]
sishen commented on HBASE-974:
------------------------------
I have patched Dan's previous work on this tool to work with 0.18/0.19.
However, looking forward to see this effort.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-974) New Backup Export/Import MR for
0.18/0.19
Posted by "sishen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644670#action_12644670 ]
sishen commented on HBASE-974:
------------------------------
I have patched Dan's previous work on this tool to work with 0.18/0.19.
However, looking forward to see this effort.
> New Backup Export/Import MR for 0.18/0.19
> -----------------------------------------
>
> Key: HBASE-974
> URL: https://issues.apache.org/jira/browse/HBASE-974
> Project: Hadoop HBase
> Issue Type: New Feature
> Affects Versions: 0.18.0, 0.18.1, 0.19.0
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Minor
>
> Erik has built a new backup tool that's compatible with newer versions of HBase.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.