MySQL/Oracle to CSV

["Ravi Papisetti (rpapiset)" <rp...@cisco.com>]

Hi,

We are trying to find right workflow for ingesting data from a relational database to csv files on hadoop.

There are few options we could think of:

  *   Write a sqoop script and wrap under ExecuteProcess
  *   Stitch a workflow using "QueryDataBaseTable" -> ConvertAVROToJson -> ReplaceTextWithMapping (looks complext to me for a simple flow)

Appreciate if you can share any experiences and best practices of using apache NiFi for this use case.



Thanks,

Ravi Papisetti

Technical Leader

Services Technology Incubation Center<http://wwwin.cisco.com/CustAdv/ts/cstg/stic/>

rpapiset@cisco.com<ma...@cisco.com>

Phone: +1 512 340 3377


[stic-logo-email-blue]

Re: MySQL/Oracle to CSV

[Manish Gupta 8 <mg...@sapient.com>]

The way we have implemented in our project (ms sql server to hadoop) is -
1. Sql Agent job runs every 5 minutes and generate a new tab separated file with incremental changes, in a network share.
2. Nifi pick up the files, merge content (suitable for hdfs), covert to avro, load in hive external tables.

This is running for around 15 tables, 3 databases.

Thanks,
Manish



On Tue, Jul 19, 2016 at 10:39 PM +0530, "Ravi Papisetti (rpapiset)" <rp...@cisco.com>> wrote:

Hi,

We are trying to find right workflow for ingesting data from a relational database to csv files on hadoop.

There are few options we could think of:

  *   Write a sqoop script and wrap under ExecuteProcess
  *   Stitch a workflow using "QueryDataBaseTable" -> ConvertAVROToJson -> ReplaceTextWithMapping (looks complext to me for a simple flow)

Appreciate if you can share any experiences and best practices of using apache NiFi for this use case.



Thanks,

Ravi Papisetti

Technical Leader

Services Technology Incubation Center<http://wwwin.cisco.com/CustAdv/ts/cstg/stic/>

rpapiset@cisco.com<ma...@cisco.com>

Phone: +1 512 340 3377


[stic-logo-email-blue]