Question on looping through a directory and emitting several files using Storm Topolgy

["Rajabhathor, Selvaraj (Contractor)" <se...@fanniemae.com>]

Hi

I am working on a POC to migrate current unix based files into HBase .

We believe we can use Storm to migrate these files - I have successfully implemented a POC to read one file and emit it across a test Topology that I dfined.
My next goal is to sort of "loop through' a directory and emit several files in a similar fashion.

How can I implement such a Topology using Storm?

Thanks
Raj


Regards,
Raj Rajabhathor
Big Data Architect,
Capco Contractor @ FannieMae<mailto:Contractor@FannieMae(917)952>
(917)952<mailto:Contractor@FannieMae(917)952>-5597 (cell)
703-833-2539 (direct)


This e-mail and its attachments are confidential and solely for the intended addressee(s). Do not share or use them without Fannie Mae's approval. If received in error, delete them and contact the sender.

Re: Question on looping through a directory and emitting several files using Storm Topolgy

[Bobby Evans <ev...@yahoo-inc.com.INVALID>]

Why use storm, unless you need to transform the data in a global way (aka streaming join or per-agragation) and the data is too big for a single box to handle, or this is going to be an ongoing process.  Just write a little app that takes a file on the command line and uploads it to HBase.  Then you can use find and xargs to upload them in parallel
find ./dir/to/data -type f | xargs -n 1 -P <NUM_IN_PARALLEL> <COMMAND_TO_UPLOAD>
If you need multiple machines to do this, then launch subsets of the data on multiple different machines.  Storm just feels like overkill for a simple upload.

- Bobby
 


     On Tuesday, June 9, 2015 11:32 AM, "Rajabhathor, Selvaraj (Contractor)" <se...@fanniemae.com> wrote:
   

 Hi

I am working on a POC to migrate current unix based files into HBase .

We believe we can use Storm to migrate these files - I have successfully implemented a POC to read one file and emit it across a test Topology that I dfined.
My next goal is to sort of "loop through' a directory and emit several files in a similar fashion.

How can I implement such a Topology using Storm?

Thanks
Raj


Regards,
Raj Rajabhathor
Big Data Architect,
Capco Contractor @ FannieMae<mailto:Contractor@FannieMae(917)952>
(917)952<mailto:Contractor@FannieMae(917)952>-5597 (cell)
703-833-2539 (direct)


This e-mail and its attachments are confidential and solely for the intended addressee(s). Do not share or use them without Fannie Mae's approval. If received in error, delete them and contact the sender.