You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2017/08/28 15:25:16 UTC
Question on Hdfs Writer
What is the reasoning behind not using the Storm provided hdfs bolts for
metron-writer? Anyone remember?
Re: Question on Hdfs Writer
Posted by Otto Fowler <ot...@gmail.com>.
Thanks!
On August 28, 2017 at 11:29:52, Casey Stella (cestella@gmail.com) wrote:
The storm metron-writer bolt has to write to many different sensor's HDFS
files. The hdfs bolts provided by storm only support writing to one file
(or set of files). The alternative would have been a bolt per sensor,
which did not support our architecture.
In short, the hdfs bolt from storm would have necessitated a bolt per
sensor and adding new sensors would have required topology restart and
recreation; this isn't something that we wanted.
On Mon, Aug 28, 2017 at 11:25 AM, Otto Fowler <ot...@gmail.com>
wrote:
> What is the reasoning behind not using the Storm provided hdfs bolts for
> metron-writer? Anyone remember?
>
Re: Question on Hdfs Writer
Posted by Casey Stella <ce...@gmail.com>.
The storm metron-writer bolt has to write to many different sensor's HDFS
files. The hdfs bolts provided by storm only support writing to one file
(or set of files). The alternative would have been a bolt per sensor,
which did not support our architecture.
In short, the hdfs bolt from storm would have necessitated a bolt per
sensor and adding new sensors would have required topology restart and
recreation; this isn't something that we wanted.
On Mon, Aug 28, 2017 at 11:25 AM, Otto Fowler <ot...@gmail.com>
wrote:
> What is the reasoning behind not using the Storm provided hdfs bolts for
> metron-writer? Anyone remember?
>