You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Meng QingPing <mq...@gmail.com> on 2014/04/28 10:42:18 UTC
Can set different dfs.replication for different dirs
Hi,
I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
dfs /user. How to configure? Both /tmp and /user are generated by
mapreduce job or hive or sqoop.
Thanks,
Jack
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Another question
Can set expired time for /tmp or set yarn/mapreduce to remove the expired
tmp files periodically?
Thanks,
Jack
2014-04-29 16:56 GMT+08:00 Meng QingPing <mq...@gmail.com>:
> Thanks for all replies.
>
> The files in /tmp most are generated by hadoop jobs. Can set the
> yarn/mapreduce to specify one repication for tmp files?
>
> Thanks,
> Jack
>
>
> 2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
>
> Hello Nitin,
>>
>> HDFS replication factor is always associated with file level. When your
>> copying or creating file for any directory will set to default replicas.
>> but you can specify your replication when creating or copying files
>> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>>
>> and in java
>>
>> FileSystem fs = FileSystem.get(new Configuration());
>> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> Sudhakar,
>>>
>>> will this set for the new files being written to those directories?
>>>
>>>
>>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>>
>>>> Changes the replication factor of a file. -R option is for recursively
>>>> increasing the replication factor of files within a directory.
>>>>
>>>> Example:
>>>>
>>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>>
>>>> hadoop dfs -setrep -R -w 1 /dir/
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> DFS replication is set on the file level (block level) or at cluster
>>>>> level (if you do not specify the replication factor while writing the file
>>>>> then this one is picked).
>>>>> As per my understanding, there is nothing for directories.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
>>>>>> 3 for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>>> mapreduce job or hive or sqoop.
>>>>>>
>>>>>> Thanks,
>>>>>> Jack
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> ...sudhakara
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Thanks,
> Qingping
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Another question
Can set expired time for /tmp or set yarn/mapreduce to remove the expired
tmp files periodically?
Thanks,
Jack
2014-04-29 16:56 GMT+08:00 Meng QingPing <mq...@gmail.com>:
> Thanks for all replies.
>
> The files in /tmp most are generated by hadoop jobs. Can set the
> yarn/mapreduce to specify one repication for tmp files?
>
> Thanks,
> Jack
>
>
> 2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
>
> Hello Nitin,
>>
>> HDFS replication factor is always associated with file level. When your
>> copying or creating file for any directory will set to default replicas.
>> but you can specify your replication when creating or copying files
>> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>>
>> and in java
>>
>> FileSystem fs = FileSystem.get(new Configuration());
>> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> Sudhakar,
>>>
>>> will this set for the new files being written to those directories?
>>>
>>>
>>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>>
>>>> Changes the replication factor of a file. -R option is for recursively
>>>> increasing the replication factor of files within a directory.
>>>>
>>>> Example:
>>>>
>>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>>
>>>> hadoop dfs -setrep -R -w 1 /dir/
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> DFS replication is set on the file level (block level) or at cluster
>>>>> level (if you do not specify the replication factor while writing the file
>>>>> then this one is picked).
>>>>> As per my understanding, there is nothing for directories.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
>>>>>> 3 for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>>> mapreduce job or hive or sqoop.
>>>>>>
>>>>>> Thanks,
>>>>>> Jack
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> ...sudhakara
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Thanks,
> Qingping
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Another question
Can set expired time for /tmp or set yarn/mapreduce to remove the expired
tmp files periodically?
Thanks,
Jack
2014-04-29 16:56 GMT+08:00 Meng QingPing <mq...@gmail.com>:
> Thanks for all replies.
>
> The files in /tmp most are generated by hadoop jobs. Can set the
> yarn/mapreduce to specify one repication for tmp files?
>
> Thanks,
> Jack
>
>
> 2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
>
> Hello Nitin,
>>
>> HDFS replication factor is always associated with file level. When your
>> copying or creating file for any directory will set to default replicas.
>> but you can specify your replication when creating or copying files
>> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>>
>> and in java
>>
>> FileSystem fs = FileSystem.get(new Configuration());
>> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> Sudhakar,
>>>
>>> will this set for the new files being written to those directories?
>>>
>>>
>>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>>
>>>> Changes the replication factor of a file. -R option is for recursively
>>>> increasing the replication factor of files within a directory.
>>>>
>>>> Example:
>>>>
>>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>>
>>>> hadoop dfs -setrep -R -w 1 /dir/
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> DFS replication is set on the file level (block level) or at cluster
>>>>> level (if you do not specify the replication factor while writing the file
>>>>> then this one is picked).
>>>>> As per my understanding, there is nothing for directories.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
>>>>>> 3 for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>>> mapreduce job or hive or sqoop.
>>>>>>
>>>>>> Thanks,
>>>>>> Jack
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> ...sudhakara
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Thanks,
> Qingping
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Another question
Can set expired time for /tmp or set yarn/mapreduce to remove the expired
tmp files periodically?
Thanks,
Jack
2014-04-29 16:56 GMT+08:00 Meng QingPing <mq...@gmail.com>:
> Thanks for all replies.
>
> The files in /tmp most are generated by hadoop jobs. Can set the
> yarn/mapreduce to specify one repication for tmp files?
>
> Thanks,
> Jack
>
>
> 2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
>
> Hello Nitin,
>>
>> HDFS replication factor is always associated with file level. When your
>> copying or creating file for any directory will set to default replicas.
>> but you can specify your replication when creating or copying files
>> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>>
>> and in java
>>
>> FileSystem fs = FileSystem.get(new Configuration());
>> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> Sudhakar,
>>>
>>> will this set for the new files being written to those directories?
>>>
>>>
>>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>>
>>>> Changes the replication factor of a file. -R option is for recursively
>>>> increasing the replication factor of files within a directory.
>>>>
>>>> Example:
>>>>
>>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>>
>>>> hadoop dfs -setrep -R -w 1 /dir/
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> DFS replication is set on the file level (block level) or at cluster
>>>>> level (if you do not specify the replication factor while writing the file
>>>>> then this one is picked).
>>>>> As per my understanding, there is nothing for directories.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
>>>>>> 3 for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>>> mapreduce job or hive or sqoop.
>>>>>>
>>>>>> Thanks,
>>>>>> Jack
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>> ...sudhakara
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Thanks,
> Qingping
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Thanks for all replies.
The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?
Thanks,
Jack
2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
> Hello Nitin,
>
> HDFS replication factor is always associated with file level. When your
> copying or creating file for any directory will set to default replicas.
> but you can specify your replication when creating or copying files
> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>
> and in java
>
> FileSystem fs = FileSystem.get(new Configuration());
> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>
>
>
>
> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> Sudhakar,
>>
>> will this set for the new files being written to those directories?
>>
>>
>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>
>>> Changes the replication factor of a file. -R option is for recursively
>>> increasing the replication factor of files within a directory.
>>>
>>> Example:
>>>
>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>
>>> hadoop dfs -setrep -R -w 1 /dir/
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> DFS replication is set on the file level (block level) or at cluster
>>>> level (if you do not specify the replication factor while writing the file
>>>> then this one is picked).
>>>> As per my understanding, there is nothing for directories.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>> mapreduce job or hive or sqoop.
>>>>>
>>>>> Thanks,
>>>>> Jack
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> ...sudhakara
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Thanks for all replies.
The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?
Thanks,
Jack
2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
> Hello Nitin,
>
> HDFS replication factor is always associated with file level. When your
> copying or creating file for any directory will set to default replicas.
> but you can specify your replication when creating or copying files
> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>
> and in java
>
> FileSystem fs = FileSystem.get(new Configuration());
> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>
>
>
>
> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> Sudhakar,
>>
>> will this set for the new files being written to those directories?
>>
>>
>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>
>>> Changes the replication factor of a file. -R option is for recursively
>>> increasing the replication factor of files within a directory.
>>>
>>> Example:
>>>
>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>
>>> hadoop dfs -setrep -R -w 1 /dir/
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> DFS replication is set on the file level (block level) or at cluster
>>>> level (if you do not specify the replication factor while writing the file
>>>> then this one is picked).
>>>> As per my understanding, there is nothing for directories.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>> mapreduce job or hive or sqoop.
>>>>>
>>>>> Thanks,
>>>>> Jack
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> ...sudhakara
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Thanks for all replies.
The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?
Thanks,
Jack
2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
> Hello Nitin,
>
> HDFS replication factor is always associated with file level. When your
> copying or creating file for any directory will set to default replicas.
> but you can specify your replication when creating or copying files
> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>
> and in java
>
> FileSystem fs = FileSystem.get(new Configuration());
> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>
>
>
>
> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> Sudhakar,
>>
>> will this set for the new files being written to those directories?
>>
>>
>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>
>>> Changes the replication factor of a file. -R option is for recursively
>>> increasing the replication factor of files within a directory.
>>>
>>> Example:
>>>
>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>
>>> hadoop dfs -setrep -R -w 1 /dir/
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> DFS replication is set on the file level (block level) or at cluster
>>>> level (if you do not specify the replication factor while writing the file
>>>> then this one is picked).
>>>> As per my understanding, there is nothing for directories.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>> mapreduce job or hive or sqoop.
>>>>>
>>>>> Thanks,
>>>>> Jack
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> ...sudhakara
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by Meng QingPing <mq...@gmail.com>.
Thanks for all replies.
The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?
Thanks,
Jack
2014-04-29 16:40 GMT+08:00 sudhakara st <su...@gmail.com>:
> Hello Nitin,
>
> HDFS replication factor is always associated with file level. When your
> copying or creating file for any directory will set to default replicas.
> but you can specify your replication when creating or copying files
> hadoop fs -D dfs.replication=2 -put foo.txt fsput
>
> and in java
>
> FileSystem fs = FileSystem.get(new Configuration());
> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>
>
>
>
> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> Sudhakar,
>>
>> will this set for the new files being written to those directories?
>>
>>
>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>>
>>> Changes the replication factor of a file. -R option is for recursively
>>> increasing the replication factor of files within a directory.
>>>
>>> Example:
>>>
>>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>
>>> hadoop dfs -setrep -R -w 1 /dir/
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> DFS replication is set on the file level (block level) or at cluster
>>>> level (if you do not specify the replication factor while writing the file
>>>> then this one is picked).
>>>> As per my understanding, there is nothing for directories.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>>> mapreduce job or hive or sqoop.
>>>>>
>>>>> Thanks,
>>>>> Jack
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> ...sudhakara
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Thanks,
Qingping
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Hello Nitin,
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
> Sudhakar,
>
> will this set for the new files being written to those directories?
>
>
> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Changes the replication factor of a file. -R option is for recursively
>> increasing the replication factor of files within a directory.
>>
>> Example:
>>
>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>
>> hadoop dfs -setrep -R -w 1 /dir/
>>
>>
>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> DFS replication is set on the file level (block level) or at cluster
>>> level (if you do not specify the replication factor while writing the file
>>> then this one is picked).
>>> As per my understanding, there is nothing for directories.
>>>
>>>
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>> mapreduce job or hive or sqoop.
>>>>
>>>> Thanks,
>>>> Jack
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Hello Nitin,
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
> Sudhakar,
>
> will this set for the new files being written to those directories?
>
>
> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Changes the replication factor of a file. -R option is for recursively
>> increasing the replication factor of files within a directory.
>>
>> Example:
>>
>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>
>> hadoop dfs -setrep -R -w 1 /dir/
>>
>>
>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> DFS replication is set on the file level (block level) or at cluster
>>> level (if you do not specify the replication factor while writing the file
>>> then this one is picked).
>>> As per my understanding, there is nothing for directories.
>>>
>>>
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>> mapreduce job or hive or sqoop.
>>>>
>>>> Thanks,
>>>> Jack
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Hello Nitin,
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
> Sudhakar,
>
> will this set for the new files being written to those directories?
>
>
> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Changes the replication factor of a file. -R option is for recursively
>> increasing the replication factor of files within a directory.
>>
>> Example:
>>
>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>
>> hadoop dfs -setrep -R -w 1 /dir/
>>
>>
>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> DFS replication is set on the file level (block level) or at cluster
>>> level (if you do not specify the replication factor while writing the file
>>> then this one is picked).
>>> As per my understanding, there is nothing for directories.
>>>
>>>
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>> mapreduce job or hive or sqoop.
>>>>
>>>> Thanks,
>>>> Jack
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Hello Nitin,
HDFS replication factor is always associated with file level. When your
copying or creating file for any directory will set to default replicas.
but you can specify your replication when creating or copying files
hadoop fs -D dfs.replication=2 -put foo.txt fsput
and in java
FileSystem fs = FileSystem.get(new Configuration());
fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <ni...@gmail.com>wrote:
> Sudhakar,
>
> will this set for the new files being written to those directories?
>
>
> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Changes the replication factor of a file. -R option is for recursively
>> increasing the replication factor of files within a directory.
>>
>> Example:
>>
>> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>
>> hadoop dfs -setrep -R -w 1 /dir/
>>
>>
>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> DFS replication is set on the file level (block level) or at cluster
>>> level (if you do not specify the replication factor while writing the file
>>> then this one is picked).
>>> As per my understanding, there is nothing for directories.
>>>
>>>
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>>> mapreduce job or hive or sqoop.
>>>>
>>>> Thanks,
>>>> Jack
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>>
>> Regards,
>> ...sudhakara
>>
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
Sudhakar,
will this set for the new files being written to those directories?
On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
> Changes the replication factor of a file. -R option is for recursively
> increasing the replication factor of files within a directory.
>
> Example:
>
> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>
> hadoop dfs -setrep -R -w 1 /dir/
>
>
> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> DFS replication is set on the file level (block level) or at cluster
>> level (if you do not specify the replication factor while writing the file
>> then this one is picked).
>> As per my understanding, there is nothing for directories.
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>> mapreduce job or hive or sqoop.
>>>
>>> Thanks,
>>> Jack
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
Sudhakar,
will this set for the new files being written to those directories?
On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
> Changes the replication factor of a file. -R option is for recursively
> increasing the replication factor of files within a directory.
>
> Example:
>
> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>
> hadoop dfs -setrep -R -w 1 /dir/
>
>
> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> DFS replication is set on the file level (block level) or at cluster
>> level (if you do not specify the replication factor while writing the file
>> then this one is picked).
>> As per my understanding, there is nothing for directories.
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>> mapreduce job or hive or sqoop.
>>>
>>> Thanks,
>>> Jack
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
Sudhakar,
will this set for the new files being written to those directories?
On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
> Changes the replication factor of a file. -R option is for recursively
> increasing the replication factor of files within a directory.
>
> Example:
>
> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>
> hadoop dfs -setrep -R -w 1 /dir/
>
>
> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> DFS replication is set on the file level (block level) or at cluster
>> level (if you do not specify the replication factor while writing the file
>> then this one is picked).
>> As per my understanding, there is nothing for directories.
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>> mapreduce job or hive or sqoop.
>>>
>>> Thanks,
>>> Jack
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
Sudhakar,
will this set for the new files being written to those directories?
On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <su...@gmail.com>wrote:
> Changes the replication factor of a file. -R option is for recursively
> increasing the replication factor of files within a directory.
>
> Example:
>
> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>
> hadoop dfs -setrep -R -w 1 /dir/
>
>
> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> DFS replication is set on the file level (block level) or at cluster
>> level (if you do not specify the replication factor while writing the file
>> then this one is picked).
>> As per my understanding, there is nothing for directories.
>>
>>
>>
>>
>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>>> for dfs /user. How to configure? Both /tmp and /user are generated by
>>> mapreduce job or hive or sqoop.
>>>
>>> Thanks,
>>> Jack
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
Example:
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
> DFS replication is set on the file level (block level) or at cluster level
> (if you do not specify the replication factor while writing the file then
> this one is picked).
> As per my understanding, there is nothing for directories.
>
>
>
>
> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>
>> Hi,
>>
>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>> for dfs /user. How to configure? Both /tmp and /user are generated by
>> mapreduce job or hive or sqoop.
>>
>> Thanks,
>> Jack
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
Example:
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
> DFS replication is set on the file level (block level) or at cluster level
> (if you do not specify the replication factor while writing the file then
> this one is picked).
> As per my understanding, there is nothing for directories.
>
>
>
>
> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>
>> Hi,
>>
>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>> for dfs /user. How to configure? Both /tmp and /user are generated by
>> mapreduce job or hive or sqoop.
>>
>> Thanks,
>> Jack
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
Example:
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
> DFS replication is set on the file level (block level) or at cluster level
> (if you do not specify the replication factor while writing the file then
> this one is picked).
> As per my understanding, there is nothing for directories.
>
>
>
>
> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>
>> Hi,
>>
>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>> for dfs /user. How to configure? Both /tmp and /user are generated by
>> mapreduce job or hive or sqoop.
>>
>> Thanks,
>> Jack
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by sudhakara st <su...@gmail.com>.
Changes the replication factor of a file. -R option is for recursively
increasing the replication factor of files within a directory.
Example:
- hadoop fs -setrep -w 3 -R /user/hadoop/dir1
hadoop dfs -setrep -R -w 1 /dir/
On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <ni...@gmail.com>wrote:
> DFS replication is set on the file level (block level) or at cluster level
> (if you do not specify the replication factor while writing the file then
> this one is picked).
> As per my understanding, there is nothing for directories.
>
>
>
>
> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com>wrote:
>
>> Hi,
>>
>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3
>> for dfs /user. How to configure? Both /tmp and /user are generated by
>> mapreduce job or hive or sqoop.
>>
>> Thanks,
>> Jack
>>
>
>
>
> --
> Nitin Pawar
>
--
Regards,
...sudhakara
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com> wrote:
> Hi,
>
> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
> dfs /user. How to configure? Both /tmp and /user are generated by
> mapreduce job or hive or sqoop.
>
> Thanks,
> Jack
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com> wrote:
> Hi,
>
> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
> dfs /user. How to configure? Both /tmp and /user are generated by
> mapreduce job or hive or sqoop.
>
> Thanks,
> Jack
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com> wrote:
> Hi,
>
> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
> dfs /user. How to configure? Both /tmp and /user are generated by
> mapreduce job or hive or sqoop.
>
> Thanks,
> Jack
>
--
Nitin Pawar
Re: Can set different dfs.replication for different dirs
Posted by Nitin Pawar <ni...@gmail.com>.
DFS replication is set on the file level (block level) or at cluster level
(if you do not specify the replication factor while writing the file then
this one is picked).
As per my understanding, there is nothing for directories.
On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mq...@gmail.com> wrote:
> Hi,
>
> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 for
> dfs /user. How to configure? Both /tmp and /user are generated by
> mapreduce job or hive or sqoop.
>
> Thanks,
> Jack
>
--
Nitin Pawar