You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Varun Sharma <va...@gmail.com> on 2015/08/19 21:20:19 UTC
Specifying replication factor & block size during distcp
Hi,
I am running a Distcp programmatically from Hadoop cluster to another -
using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
replication factor for my files. How can I achieve that ?
Thanks !
Varun
Re: Specifying replication factor & block size during distcp
Posted by nataraj jonnalagadda <na...@gmail.com>.
Not sure if this feature is available. A workaround would be to update
replication factor and block size at the HDFS level and reverting the
changes after the distcp is complete. This is good for a one time copy. :-)
On Wed, Aug 19, 2015 at 12:52 PM, Ted Yu <yu...@gmail.com> wrote:
> I looked at
> hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
> but didn't find such switch.
>
> FYI
>
> On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running a Distcp programmatically from Hadoop cluster to another -
>> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
>> replication factor for my files. How can I achieve that ?
>>
>> Thanks !
>> Varun
>>
>
>
Re: Specifying replication factor & block size during distcp
Posted by nataraj jonnalagadda <na...@gmail.com>.
Not sure if this feature is available. A workaround would be to update
replication factor and block size at the HDFS level and reverting the
changes after the distcp is complete. This is good for a one time copy. :-)
On Wed, Aug 19, 2015 at 12:52 PM, Ted Yu <yu...@gmail.com> wrote:
> I looked at
> hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
> but didn't find such switch.
>
> FYI
>
> On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running a Distcp programmatically from Hadoop cluster to another -
>> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
>> replication factor for my files. How can I achieve that ?
>>
>> Thanks !
>> Varun
>>
>
>
Re: Specifying replication factor & block size during distcp
Posted by nataraj jonnalagadda <na...@gmail.com>.
Not sure if this feature is available. A workaround would be to update
replication factor and block size at the HDFS level and reverting the
changes after the distcp is complete. This is good for a one time copy. :-)
On Wed, Aug 19, 2015 at 12:52 PM, Ted Yu <yu...@gmail.com> wrote:
> I looked at
> hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
> but didn't find such switch.
>
> FYI
>
> On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running a Distcp programmatically from Hadoop cluster to another -
>> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
>> replication factor for my files. How can I achieve that ?
>>
>> Thanks !
>> Varun
>>
>
>
Re: Specifying replication factor & block size during distcp
Posted by nataraj jonnalagadda <na...@gmail.com>.
Not sure if this feature is available. A workaround would be to update
replication factor and block size at the HDFS level and reverting the
changes after the distcp is complete. This is good for a one time copy. :-)
On Wed, Aug 19, 2015 at 12:52 PM, Ted Yu <yu...@gmail.com> wrote:
> I looked at
> hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
> but didn't find such switch.
>
> FYI
>
> On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running a Distcp programmatically from Hadoop cluster to another -
>> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
>> replication factor for my files. How can I achieve that ?
>>
>> Thanks !
>> Varun
>>
>
>
Re: Specifying replication factor & block size during distcp
Posted by Ted Yu <yu...@gmail.com>.
I looked at
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
but didn't find such switch.
FYI
On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com> wrote:
> Hi,
>
> I am running a Distcp programmatically from Hadoop cluster to another -
> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
> replication factor for my files. How can I achieve that ?
>
> Thanks !
> Varun
>
Re: Specifying replication factor & block size during distcp
Posted by Ted Yu <yu...@gmail.com>.
I looked at
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
but didn't find such switch.
FYI
On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com> wrote:
> Hi,
>
> I am running a Distcp programmatically from Hadoop cluster to another -
> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
> replication factor for my files. How can I achieve that ?
>
> Thanks !
> Varun
>
Re: Specifying replication factor & block size during distcp
Posted by Ted Yu <yu...@gmail.com>.
I looked at
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
but didn't find such switch.
FYI
On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com> wrote:
> Hi,
>
> I am running a Distcp programmatically from Hadoop cluster to another -
> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
> replication factor for my files. How can I achieve that ?
>
> Thanks !
> Varun
>
Re: Specifying replication factor & block size during distcp
Posted by Ted Yu <yu...@gmail.com>.
I looked at
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
but didn't find such switch.
FYI
On Wed, Aug 19, 2015 at 12:20 PM, Varun Sharma <va...@gmail.com> wrote:
> Hi,
>
> I am running a Distcp programmatically from Hadoop cluster to another -
> using Hadoop 2.7 and distcp v2. I would like to set a custom block size and
> replication factor for my files. How can I achieve that ?
>
> Thanks !
> Varun
>