You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2013/09/19 14:40:02 UTC
HDFs file-create performance
Are there any rough numbers one can give me regarding the latency of creating, writing, and closing a small HDFS-based file? Does replication have a big impact? I am trying to decide whether to communicate some modestly-sized (~200KB) information via HDFS files or go to the trouble of creating a protocol.
Thanks
John
Re: HDFs file-create performance
Posted by "M. C. Srivas" <mc...@gmail.com>.
Small file creation is a well-documented major problem (and bottleneck) in
HDFS. You can either roll your own protocol, or use MapR which is about
100x faster and 1000x scalable than HDFS for this particular problem.
Re: HDFs file-create performance
Posted by "M. C. Srivas" <mc...@gmail.com>.
Small file creation is a well-documented major problem (and bottleneck) in
HDFS. You can either roll your own protocol, or use MapR which is about
100x faster and 1000x scalable than HDFS for this particular problem.
Re: HDFs file-create performance
Posted by "M. C. Srivas" <mc...@gmail.com>.
Small file creation is a well-documented major problem (and bottleneck) in
HDFS. You can either roll your own protocol, or use MapR which is about
100x faster and 1000x scalable than HDFS for this particular problem.
Re: HDFs file-create performance
Posted by "M. C. Srivas" <mc...@gmail.com>.
Small file creation is a well-documented major problem (and bottleneck) in
HDFS. You can either roll your own protocol, or use MapR which is about
100x faster and 1000x scalable than HDFS for this particular problem.