You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by prem yadav <ip...@gmail.com> on 2012/08/08 15:20:46 UTC

Loading data from S3

Hi,
I recently used a backup tool to back up all my HDFS data to S3. The data
is on S3 in multiparts.
I need to test the restore now. Could you please give me some pointers on
how to test this.

1) Do I need to create another cluster? The data is around 3 TB in size.
2) How do I upload multipart data from S3 to HDFS cluster?


regards,
Prem

Re: Loading data from S3

Posted by Mohammad Tariq <do...@gmail.com>.

Why all the unsubscription requests are coming to this address??

Regards,
    Mohammad Tariq


On Wed, Aug 8, 2012 at 9:08 PM, Wil Moore III <wi...@wilmoore.com> wrote:
> unsubscribe

Re: Loading data from S3

Posted by Mohammad Tariq <do...@gmail.com>.

Why all the unsubscription requests are coming to this address??

Regards,
    Mohammad Tariq


On Wed, Aug 8, 2012 at 9:08 PM, Wil Moore III <wi...@wilmoore.com> wrote:
> unsubscribe

Re: Loading data from S3

Posted by Mohammad Tariq <do...@gmail.com>.

Why all the unsubscription requests are coming to this address??

Regards,
    Mohammad Tariq


On Wed, Aug 8, 2012 at 9:08 PM, Wil Moore III <wi...@wilmoore.com> wrote:
> unsubscribe

Re: Loading data from S3

Posted by Mohammad Tariq <do...@gmail.com>.

Why all the unsubscription requests are coming to this address??

Regards,
    Mohammad Tariq


On Wed, Aug 8, 2012 at 9:08 PM, Wil Moore III <wi...@wilmoore.com> wrote:
> unsubscribe

Re: Loading data from S3

Posted by Wil Moore III <wi...@wilmoore.com>.

unsubscribe

Re: Loading data from S3

Posted by prem yadav <ip...@gmail.com>.

I have used the tool Hbackup from https://github.com/urbanairship/hbackup

I will look into S3distcp. The name suggests ot should be sufficient for me
to load the data.
However I have a more generic question. How do people who backup the Hbase
data tables to S3 test the restore.

My backup ran for about a day and there were a couple of exceptions in the
logs. How do I test the table? Do I need to recreate the hadoop/Hbase
cluster and test whether everything went well?

regards,
Prem
On Wed, Aug 8, 2012 at 6:54 PM, Dan Young <da...@gmail.com> wrote:

> Have you looked into s3distcp ?
>
> Regards ,
>
> Dano
> On Aug 8, 2012 7:21 AM, "prem yadav" <ip...@gmail.com> wrote:
>
>> Hi,
>> I recently used a backup tool to back up all my HDFS data to S3. The data
>> is on S3 in multiparts.
>> I need to test the restore now. Could you please give me some pointers on
>> how to test this.
>>
>> 1) Do I need to create another cluster? The data is around 3 TB in size.
>> 2) How do I upload multipart data from S3 to HDFS cluster?
>>
>>
>> regards,
>> Prem
>>
>>

Re: Loading data from S3

Posted by prem yadav <ip...@gmail.com>.

I have used the tool Hbackup from https://github.com/urbanairship/hbackup

I will look into S3distcp. The name suggests ot should be sufficient for me
to load the data.
However I have a more generic question. How do people who backup the Hbase
data tables to S3 test the restore.

My backup ran for about a day and there were a couple of exceptions in the
logs. How do I test the table? Do I need to recreate the hadoop/Hbase
cluster and test whether everything went well?

regards,
Prem
On Wed, Aug 8, 2012 at 6:54 PM, Dan Young <da...@gmail.com> wrote:

> Have you looked into s3distcp ?
>
> Regards ,
>
> Dano
> On Aug 8, 2012 7:21 AM, "prem yadav" <ip...@gmail.com> wrote:
>
>> Hi,
>> I recently used a backup tool to back up all my HDFS data to S3. The data
>> is on S3 in multiparts.
>> I need to test the restore now. Could you please give me some pointers on
>> how to test this.
>>
>> 1) Do I need to create another cluster? The data is around 3 TB in size.
>> 2) How do I upload multipart data from S3 to HDFS cluster?
>>
>>
>> regards,
>> Prem
>>
>>

Unsubscribe

Posted by "Zhao, Frank (usd)" <fr...@emc.com>.

unsubscribe

On 2012-8-8, at 22:06, "Browning,Jeremy" <br...@oclc.org>> wrote:

unsubscribe

From: Chinni, Ravi [mailto:rchinni@syncsort.com]
Sent: Wednesday, August 08, 2012 10:03 AM
To: <ma...@hadoop.apache.org> user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: unsubscribe

unsubscribe

________________________________

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

Unsubscribe

Posted by "Zhao, Frank (usd)" <fr...@emc.com>.

unsubscribe

On 2012-8-8, at 22:06, "Browning,Jeremy" <br...@oclc.org>> wrote:

unsubscribe

From: Chinni, Ravi [mailto:rchinni@syncsort.com]
Sent: Wednesday, August 08, 2012 10:03 AM
To: <ma...@hadoop.apache.org> user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: unsubscribe

unsubscribe

________________________________

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

Unsubscribe

Posted by "Zhao, Frank (usd)" <fr...@emc.com>.

unsubscribe

On 2012-8-8, at 22:06, "Browning,Jeremy" <br...@oclc.org>> wrote:

unsubscribe

From: Chinni, Ravi [mailto:rchinni@syncsort.com]
Sent: Wednesday, August 08, 2012 10:03 AM
To: <ma...@hadoop.apache.org> user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: unsubscribe

unsubscribe

________________________________

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

Unsubscribe

Posted by "Zhao, Frank (usd)" <fr...@emc.com>.

unsubscribe

On 2012-8-8, at 22:06, "Browning,Jeremy" <br...@oclc.org>> wrote:

unsubscribe

From: Chinni, Ravi [mailto:rchinni@syncsort.com]
Sent: Wednesday, August 08, 2012 10:03 AM
To: <ma...@hadoop.apache.org> user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: unsubscribe

unsubscribe

________________________________

ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

RE: unsubscribe

Posted by "Browning,Jeremy" <br...@oclc.org>.

unsubscribe

From: Chinni, Ravi [mailto:rchinni@syncsort.com] 
Sent: Wednesday, August 08, 2012 10:03 AM
To: user@hadoop.apache.org
Subject: unsubscribe

unsubscribe

________________________________

ATTENTION: -----

The information contained in this message (including any files
transmitted with this message) may contain proprietary, trade secret or
other confidential and/or legally privileged information. Any pricing
information contained in this message or in any files transmitted with
this message is always confidential and cannot be shared with any third
parties without prior written approval from Syncsort. This message is
intended to be read only by the individual or entity to whom it is
addressed or by their designee. If the reader of this message is not the
intended recipient, you are on notice that any use, disclosure, copying
or distribution of this message, in any form, is strictly prohibited. If
you have received this message in error, please immediately notify the
sender and/or Syncsort and destroy all copies of this message in your
possession, custody or control.