You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by David Robison <da...@psgglobal.net> on 2016/04/15 14:49:18 UTC
Question appending data using WebHDFS
I am trying to use WebHDFS to append a large amount of data to a file. From the documentation I see that I send the op=APPEND to the NameNode first and then it sends me the location of the DataNode. Then I can start sending data to the DataNode using the op=APPEND. My question is, can I continue to write data to the DataNode or do I need to periodically recheck with the NameNode to see if it wants to send the new data to another DataNode? In other words, once I have received the DataNode's location can I continue to send as much data as I want to that DataNode or will I need at some point need to start sending data to another DataNode for the same file? I hope this is clear as I am new to HDFS. Thanks, David
Re: Question appending data using WebHDFS
Posted by Jon Ernster <jo...@gmail.com>.
Your file will be split up across numerous data nodes assuming the
file you're storing is larger than the configured block size.
Therefore you'll have file / blocksize across X data nodes. This is
all handled automatically so you don't really have the ability to say
"I want this entire file on this data node".
On Fri, Apr 15, 2016 at 6:49 AM, David Robison
<da...@psgglobal.net> wrote:
> I am trying to use WebHDFS to append a large amount of data to a file. From
> the documentation I see that I send the op=APPEND to the NameNode first and
> then it sends me the location of the DataNode. Then I can start sending data
> to the DataNode using the op=APPEND. My question is, can I continue to write
> data to the DataNode or do I need to periodically recheck with the NameNode
> to see if it wants to send the new data to another DataNode? In other words,
> once I have received the DataNode’s location can I continue to send as much
> data as I want to that DataNode or will I need at some point need to start
> sending data to another DataNode for the same file? I hope this is clear as
> I am new to HDFS. Thanks, David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Question appending data using WebHDFS
Posted by Jon Ernster <jo...@gmail.com>.
Your file will be split up across numerous data nodes assuming the
file you're storing is larger than the configured block size.
Therefore you'll have file / blocksize across X data nodes. This is
all handled automatically so you don't really have the ability to say
"I want this entire file on this data node".
On Fri, Apr 15, 2016 at 6:49 AM, David Robison
<da...@psgglobal.net> wrote:
> I am trying to use WebHDFS to append a large amount of data to a file. From
> the documentation I see that I send the op=APPEND to the NameNode first and
> then it sends me the location of the DataNode. Then I can start sending data
> to the DataNode using the op=APPEND. My question is, can I continue to write
> data to the DataNode or do I need to periodically recheck with the NameNode
> to see if it wants to send the new data to another DataNode? In other words,
> once I have received the DataNode’s location can I continue to send as much
> data as I want to that DataNode or will I need at some point need to start
> sending data to another DataNode for the same file? I hope this is clear as
> I am new to HDFS. Thanks, David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Question appending data using WebHDFS
Posted by Jon Ernster <jo...@gmail.com>.
Your file will be split up across numerous data nodes assuming the
file you're storing is larger than the configured block size.
Therefore you'll have file / blocksize across X data nodes. This is
all handled automatically so you don't really have the ability to say
"I want this entire file on this data node".
On Fri, Apr 15, 2016 at 6:49 AM, David Robison
<da...@psgglobal.net> wrote:
> I am trying to use WebHDFS to append a large amount of data to a file. From
> the documentation I see that I send the op=APPEND to the NameNode first and
> then it sends me the location of the DataNode. Then I can start sending data
> to the DataNode using the op=APPEND. My question is, can I continue to write
> data to the DataNode or do I need to periodically recheck with the NameNode
> to see if it wants to send the new data to another DataNode? In other words,
> once I have received the DataNode’s location can I continue to send as much
> data as I want to that DataNode or will I need at some point need to start
> sending data to another DataNode for the same file? I hope this is clear as
> I am new to HDFS. Thanks, David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Question appending data using WebHDFS
Posted by Jon Ernster <jo...@gmail.com>.
Your file will be split up across numerous data nodes assuming the
file you're storing is larger than the configured block size.
Therefore you'll have file / blocksize across X data nodes. This is
all handled automatically so you don't really have the ability to say
"I want this entire file on this data node".
On Fri, Apr 15, 2016 at 6:49 AM, David Robison
<da...@psgglobal.net> wrote:
> I am trying to use WebHDFS to append a large amount of data to a file. From
> the documentation I see that I send the op=APPEND to the NameNode first and
> then it sends me the location of the DataNode. Then I can start sending data
> to the DataNode using the op=APPEND. My question is, can I continue to write
> data to the DataNode or do I need to periodically recheck with the NameNode
> to see if it wants to send the new data to another DataNode? In other words,
> once I have received the DataNode’s location can I continue to send as much
> data as I want to that DataNode or will I need at some point need to start
> sending data to another DataNode for the same file? I hope this is clear as
> I am new to HDFS. Thanks, David
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org