You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by donal0412 <do...@gmail.com> on 2011/11/08 08:32:29 UTC
dfs.write.packet.size set to 2G
Hi,
I want to store lots of files in HDFS, the file size is <= 2G.
I don't want the file to split into blocks,because I need the whole file
while processing it, and I don't want to transfer blocks to one node
when processing it.
A easy way to do this would be set dfs.write.packet.size to 2G, I wonder
if some one has similar experiences or known whether this is practicable.
Will there be performance problems when set the packet size to a big number?
Thanks!
donal
Re: dfs.write.packet.size set to 2G
Posted by Ted Dunning <td...@maprtech.com>.
By snapshots, I mean that you can freeze a copy of a portion of the the
file system for later use as a backup or reference. By mirror, I mean that
a snapshot can be transported to another location in the same cluster or to
another cluster and the mirrored image will be updated atomically to the
new state.
See http://mapr.com/products/why-mapr for more info.
On Tue, Nov 8, 2011 at 3:42 AM, donal0412 <do...@gmail.com> wrote:
> Thanks! That's exactly what I want.
> And Ted, what do you mean by "snapshots and mirrors" ?
> On 2011/11/8 16:21, Harsh J wrote:
>
>> Block sizes are per-file, not permanently set on the HDFS. So create
>> your files with a sufficiently large block size (2G is OK if it fits
>> your usecase well). This way you won't have block splits, as you
>> desire.
>>
>> For example, to upload a file via the shell with a tweaked blocksize, I'd
>> do:
>>
>> hadoop dfs -Ddfs.block.size=2147483648 -copyFromLocal localFile
>> remoteFile
>>
>> Packet sizes are not what you want to tweak here.
>>
>> On Tue, Nov 8, 2011 at 1:02 PM, donal0412<do...@gmail.com> wrote:
>>
>>> Hi,
>>> I want to store lots of files in HDFS, the file size is<= 2G.
>>> I don't want the file to split into blocks,because I need the whole file
>>> while processing it, and I don't want to transfer blocks to one node when
>>> processing it.
>>> A easy way to do this would be set dfs.write.packet.size to 2G, I wonder
>>> if
>>> some one has similar experiences or known whether this is practicable.
>>> Will there be performance problems when set the packet size to a big
>>> number?
>>>
>>> Thanks!
>>> donal
>>>
>>>
>>
>>
>
Re: dfs.write.packet.size set to 2G
Posted by donal0412 <do...@gmail.com>.
Thanks! That's exactly what I want.
And Ted, what do you mean by "snapshots and mirrors" ?
On 2011/11/8 16:21, Harsh J wrote:
> Block sizes are per-file, not permanently set on the HDFS. So create
> your files with a sufficiently large block size (2G is OK if it fits
> your usecase well). This way you won't have block splits, as you
> desire.
>
> For example, to upload a file via the shell with a tweaked blocksize, I'd do:
>
> hadoop dfs -Ddfs.block.size=2147483648 -copyFromLocal localFile remoteFile
>
> Packet sizes are not what you want to tweak here.
>
> On Tue, Nov 8, 2011 at 1:02 PM, donal0412<do...@gmail.com> wrote:
>> Hi,
>> I want to store lots of files in HDFS, the file size is<= 2G.
>> I don't want the file to split into blocks,because I need the whole file
>> while processing it, and I don't want to transfer blocks to one node when
>> processing it.
>> A easy way to do this would be set dfs.write.packet.size to 2G, I wonder if
>> some one has similar experiences or known whether this is practicable.
>> Will there be performance problems when set the packet size to a big number?
>>
>> Thanks!
>> donal
>>
>
>
Re: dfs.write.packet.size set to 2G
Posted by Harsh J <ha...@cloudera.com>.
Block sizes are per-file, not permanently set on the HDFS. So create
your files with a sufficiently large block size (2G is OK if it fits
your usecase well). This way you won't have block splits, as you
desire.
For example, to upload a file via the shell with a tweaked blocksize, I'd do:
hadoop dfs -Ddfs.block.size=2147483648 -copyFromLocal localFile remoteFile
Packet sizes are not what you want to tweak here.
On Tue, Nov 8, 2011 at 1:02 PM, donal0412 <do...@gmail.com> wrote:
> Hi,
> I want to store lots of files in HDFS, the file size is <= 2G.
> I don't want the file to split into blocks,because I need the whole file
> while processing it, and I don't want to transfer blocks to one node when
> processing it.
> A easy way to do this would be set dfs.write.packet.size to 2G, I wonder if
> some one has similar experiences or known whether this is practicable.
> Will there be performance problems when set the packet size to a big number?
>
> Thanks!
> donal
>
--
Harsh J
Re: dfs.write.packet.size set to 2G
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
----- Original Message -----
From: donal0412 <do...@gmail.com>
Date: Tuesday, November 8, 2011 1:04 pm
Subject: dfs.write.packet.size set to 2G
To: hdfs-user@hadoop.apache.org
> Hi,
> I want to store lots of files in HDFS, the file size is <= 2G.
> I don't want the file to split into blocks,because I need the
> whole file
> while processing it, and I don't want to transfer blocks to one
> node
> when processing it.
> A easy way to do this would be set dfs.write.packet.size to 2G, I
> wonder
> if some one has similar experiences or known whether this is
> practicable.Will there be performance problems when set the packet
> size to a big number?
Looks you are looking at wrong configuration for your case. If you dont want to split the file, you need to increase dfs.blocksize.
In DFS data transefer will happen packet by packet. dfs.write.packet.size will represents waht is the size for this packet. So, block will be splitted into packets at client side and maintain in dataqueue. DataStreamet thread will pick the packets and transfer to DN till the block size reaches. Once it reaches the block boundary, it will close the block streams.
BTW, how you are going to process the data here? You are not going to use mapreduce for processing you Data?
>
> Thanks!
> donal
>
Regards,
Uma