You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by uday chopra <ud...@gmail.com> on 2012/11/06 01:19:18 UTC
backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as
netbackup with datadomain D-2-D appliance for doing backups of data in
hdfs? If so, what is the average dedup ratio? (I understand mileage can
vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by uday chopra <ud...@gmail.com>.
Thanks for all the responses. This is very useful information.
On Mon, Nov 5, 2012 at 11:40 PM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> I second this proposed solution. Distcp work very well with backing up
> data on the separate cluster
>
> From: Bharath Mundlapudi <bharathwork@yahoo.com<mailto:
> bharathwork@yahoo.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>, Bharath
> Mundlapudi <bh...@yahoo.com>>
> Date: Tuesday, November 6, 2012 7:10 AM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: backup of hdfs data
>
> If data is less in your cluster (say less than few GBs) then answer is
> yes. But it is an expensive route. For large data sets, traditional means
> is not feasible and it is expensive.
> If you want optimal cost based solution, you could setup another
> local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk
> is cheap :).
>
> -Bharath
>
>
> ________________________________
> From: uday chopra <uday.chopra.1970@gmail.com<mailto:
> uday.chopra.1970@gmail.com>>
> To: user@hadoop.apache.org<ma...@hadoop.apache.org>
> Sent: Monday, November 5, 2012 4:19 PM
> Subject: backup of hdfs data
>
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as
> netbackup with datadomain D-2-D appliance for doing backups of data in
> hdfs? If so, what is the average dedup ratio? (I understand mileage can
> vary based on the type of data)
>
> Thanks,
> Uday
>
>
>
Re: backup of hdfs data
Posted by uday chopra <ud...@gmail.com>.
Thanks for all the responses. This is very useful information.
On Mon, Nov 5, 2012 at 11:40 PM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> I second this proposed solution. Distcp work very well with backing up
> data on the separate cluster
>
> From: Bharath Mundlapudi <bharathwork@yahoo.com<mailto:
> bharathwork@yahoo.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>, Bharath
> Mundlapudi <bh...@yahoo.com>>
> Date: Tuesday, November 6, 2012 7:10 AM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: backup of hdfs data
>
> If data is less in your cluster (say less than few GBs) then answer is
> yes. But it is an expensive route. For large data sets, traditional means
> is not feasible and it is expensive.
> If you want optimal cost based solution, you could setup another
> local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk
> is cheap :).
>
> -Bharath
>
>
> ________________________________
> From: uday chopra <uday.chopra.1970@gmail.com<mailto:
> uday.chopra.1970@gmail.com>>
> To: user@hadoop.apache.org<ma...@hadoop.apache.org>
> Sent: Monday, November 5, 2012 4:19 PM
> Subject: backup of hdfs data
>
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as
> netbackup with datadomain D-2-D appliance for doing backups of data in
> hdfs? If so, what is the average dedup ratio? (I understand mileage can
> vary based on the type of data)
>
> Thanks,
> Uday
>
>
>
Re: backup of hdfs data
Posted by uday chopra <ud...@gmail.com>.
Thanks for all the responses. This is very useful information.
On Mon, Nov 5, 2012 at 11:40 PM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> I second this proposed solution. Distcp work very well with backing up
> data on the separate cluster
>
> From: Bharath Mundlapudi <bharathwork@yahoo.com<mailto:
> bharathwork@yahoo.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>, Bharath
> Mundlapudi <bh...@yahoo.com>>
> Date: Tuesday, November 6, 2012 7:10 AM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: backup of hdfs data
>
> If data is less in your cluster (say less than few GBs) then answer is
> yes. But it is an expensive route. For large data sets, traditional means
> is not feasible and it is expensive.
> If you want optimal cost based solution, you could setup another
> local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk
> is cheap :).
>
> -Bharath
>
>
> ________________________________
> From: uday chopra <uday.chopra.1970@gmail.com<mailto:
> uday.chopra.1970@gmail.com>>
> To: user@hadoop.apache.org<ma...@hadoop.apache.org>
> Sent: Monday, November 5, 2012 4:19 PM
> Subject: backup of hdfs data
>
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as
> netbackup with datadomain D-2-D appliance for doing backups of data in
> hdfs? If so, what is the average dedup ratio? (I understand mileage can
> vary based on the type of data)
>
> Thanks,
> Uday
>
>
>
Re: backup of hdfs data
Posted by uday chopra <ud...@gmail.com>.
Thanks for all the responses. This is very useful information.
On Mon, Nov 5, 2012 at 11:40 PM, Serge Blazhiyevskyy <
Serge.Blazhiyevskyy@nice.com> wrote:
> I second this proposed solution. Distcp work very well with backing up
> data on the separate cluster
>
> From: Bharath Mundlapudi <bharathwork@yahoo.com<mailto:
> bharathwork@yahoo.com>>
> Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>, Bharath
> Mundlapudi <bh...@yahoo.com>>
> Date: Tuesday, November 6, 2012 7:10 AM
> To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <
> user@hadoop.apache.org<ma...@hadoop.apache.org>>
> Subject: Re: backup of hdfs data
>
> If data is less in your cluster (say less than few GBs) then answer is
> yes. But it is an expensive route. For large data sets, traditional means
> is not feasible and it is expensive.
> If you want optimal cost based solution, you could setup another
> local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk
> is cheap :).
>
> -Bharath
>
>
> ________________________________
> From: uday chopra <uday.chopra.1970@gmail.com<mailto:
> uday.chopra.1970@gmail.com>>
> To: user@hadoop.apache.org<ma...@hadoop.apache.org>
> Sent: Monday, November 5, 2012 4:19 PM
> Subject: backup of hdfs data
>
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as
> netbackup with datadomain D-2-D appliance for doing backups of data in
> hdfs? If so, what is the average dedup ratio? (I understand mileage can
> vary based on the type of data)
>
> Thanks,
> Uday
>
>
>
Re: backup of hdfs data
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
I second this proposed solution. Distcp work very well with backing up data on the separate cluster
From: Bharath Mundlapudi <bh...@yahoo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Bharath Mundlapudi <bh...@yahoo.com>>
Date: Tuesday, November 6, 2012 7:10 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: backup of hdfs data
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
I second this proposed solution. Distcp work very well with backing up data on the separate cluster
From: Bharath Mundlapudi <bh...@yahoo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Bharath Mundlapudi <bh...@yahoo.com>>
Date: Tuesday, November 6, 2012 7:10 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: backup of hdfs data
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
I second this proposed solution. Distcp work very well with backing up data on the separate cluster
From: Bharath Mundlapudi <bh...@yahoo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Bharath Mundlapudi <bh...@yahoo.com>>
Date: Tuesday, November 6, 2012 7:10 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: backup of hdfs data
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Serge Blazhiyevskyy <Se...@nice.com>.
I second this proposed solution. Distcp work very well with backing up data on the separate cluster
From: Bharath Mundlapudi <bh...@yahoo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Bharath Mundlapudi <bh...@yahoo.com>>
Date: Tuesday, November 6, 2012 7:10 AM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: backup of hdfs data
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>>
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Michael Segel <mi...@hotmail.com>.
You have other options.
You could create a secondary cluster.
You could also look in to Cleversafe and what they are doing with Hadoop.
Here's the sad thing about backing up to tape... you can dump a couple of 10's of TB to tape.
You lose your system. How long will it take to recover?
And that's the thing, You need to think about this not just in DR but in terms of BCP (Business Continuity Processing)
On Nov 5, 2012, at 7:21 PM, Jay Vyas <ja...@gmail.com> wrote:
> Amazon has a really cheap, large scale backup solution called glacier which is good if your just backing up for the sake of archival in emergencies. If you need the archival to be performant, than you might want to just consider a higher replication rate.
Re: backup of hdfs data
Posted by Michael Segel <mi...@hotmail.com>.
You have other options.
You could create a secondary cluster.
You could also look in to Cleversafe and what they are doing with Hadoop.
Here's the sad thing about backing up to tape... you can dump a couple of 10's of TB to tape.
You lose your system. How long will it take to recover?
And that's the thing, You need to think about this not just in DR but in terms of BCP (Business Continuity Processing)
On Nov 5, 2012, at 7:21 PM, Jay Vyas <ja...@gmail.com> wrote:
> Amazon has a really cheap, large scale backup solution called glacier which is good if your just backing up for the sake of archival in emergencies. If you need the archival to be performant, than you might want to just consider a higher replication rate.
Re: backup of hdfs data
Posted by Michael Segel <mi...@hotmail.com>.
You have other options.
You could create a secondary cluster.
You could also look in to Cleversafe and what they are doing with Hadoop.
Here's the sad thing about backing up to tape... you can dump a couple of 10's of TB to tape.
You lose your system. How long will it take to recover?
And that's the thing, You need to think about this not just in DR but in terms of BCP (Business Continuity Processing)
On Nov 5, 2012, at 7:21 PM, Jay Vyas <ja...@gmail.com> wrote:
> Amazon has a really cheap, large scale backup solution called glacier which is good if your just backing up for the sake of archival in emergencies. If you need the archival to be performant, than you might want to just consider a higher replication rate.
Re: backup of hdfs data
Posted by Michael Segel <mi...@hotmail.com>.
You have other options.
You could create a secondary cluster.
You could also look in to Cleversafe and what they are doing with Hadoop.
Here's the sad thing about backing up to tape... you can dump a couple of 10's of TB to tape.
You lose your system. How long will it take to recover?
And that's the thing, You need to think about this not just in DR but in terms of BCP (Business Continuity Processing)
On Nov 5, 2012, at 7:21 PM, Jay Vyas <ja...@gmail.com> wrote:
> Amazon has a really cheap, large scale backup solution called glacier which is good if your just backing up for the sake of archival in emergencies. If you need the archival to be performant, than you might want to just consider a higher replication rate.
Re: backup of hdfs data
Posted by Jay Vyas <ja...@gmail.com>.
Amazon has a really cheap, large scale backup solution called glacier which
is good if your just backing up for the sake of archival in emergencies.
If you need the archival to be performant, than you might want to just
consider a higher replication rate.
Re: backup of hdfs data
Posted by Jay Vyas <ja...@gmail.com>.
Amazon has a really cheap, large scale backup solution called glacier which
is good if your just backing up for the sake of archival in emergencies.
If you need the archival to be performant, than you might want to just
consider a higher replication rate.
Re: backup of hdfs data
Posted by Jay Vyas <ja...@gmail.com>.
Amazon has a really cheap, large scale backup solution called glacier which
is good if your just backing up for the sake of archival in emergencies.
If you need the archival to be performant, than you might want to just
consider a higher replication rate.
Re: backup of hdfs data
Posted by Jay Vyas <ja...@gmail.com>.
Amazon has a really cheap, large scale backup solution called glacier which
is good if your just backing up for the sake of archival in emergencies.
If you need the archival to be performant, than you might want to just
consider a higher replication rate.
Re: backup of hdfs data
Posted by Ted Dunning <td...@maprtech.com>.
Conventional enterprise backup systems are rarely scaled for hadoop needs. Both bandwidth and size are typically lacking.
My employer, Mapr, offers a hadoop-derived distribution that includes both point in time snapshots and remote mirrors. Contact me off line for more info.
Sent from my iPhone
On Nov 5, 2012, at 4:19 PM, uday chopra <ud...@gmail.com> wrote:
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
>
> Thanks,
> Uday
Re: backup of hdfs data
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
If data is less in your cluster (say less than few GBs) then answer is yes. But it is an expensive route. For large data sets, traditional means is not feasible and it is expensive.
If you want optimal cost based solution, you could setup another local/remote cluster and try discp or simply copy hdfs files to JBODs. Disk is cheap :).
-Bharath
________________________________
From: uday chopra <ud...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, November 5, 2012 4:19 PM
Subject: backup of hdfs data
What do folks do to backup hdfs data?
Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
Thanks,
Uday
Re: backup of hdfs data
Posted by Ted Dunning <td...@maprtech.com>.
Conventional enterprise backup systems are rarely scaled for hadoop needs. Both bandwidth and size are typically lacking.
My employer, Mapr, offers a hadoop-derived distribution that includes both point in time snapshots and remote mirrors. Contact me off line for more info.
Sent from my iPhone
On Nov 5, 2012, at 4:19 PM, uday chopra <ud...@gmail.com> wrote:
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
>
> Thanks,
> Uday
Re: backup of hdfs data
Posted by Ted Dunning <td...@maprtech.com>.
Conventional enterprise backup systems are rarely scaled for hadoop needs. Both bandwidth and size are typically lacking.
My employer, Mapr, offers a hadoop-derived distribution that includes both point in time snapshots and remote mirrors. Contact me off line for more info.
Sent from my iPhone
On Nov 5, 2012, at 4:19 PM, uday chopra <ud...@gmail.com> wrote:
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
>
> Thanks,
> Uday
Re: backup of hdfs data
Posted by Ted Dunning <td...@maprtech.com>.
Conventional enterprise backup systems are rarely scaled for hadoop needs. Both bandwidth and size are typically lacking.
My employer, Mapr, offers a hadoop-derived distribution that includes both point in time snapshots and remote mirrors. Contact me off line for more info.
Sent from my iPhone
On Nov 5, 2012, at 4:19 PM, uday chopra <ud...@gmail.com> wrote:
> What do folks do to backup hdfs data?
> Has anyone experience in trying to use enterprise solutions such as netbackup with datadomain D-2-D appliance for doing backups of data in hdfs? If so, what is the average dedup ratio? (I understand mileage can vary based on the type of data)
>
> Thanks,
> Uday