Data Ingestion pipeline - NiFi to ElasticSearch

[Divya Gehlot <di...@gmail.com>]

Hi,
I am working on Data Ingestion Pipeline  using NiFi   ingest data to
Elastic Search.  Tech Stack :
Apache NiFi - 1.9.2
ElasticSearch 7.1.0

I am able to connect to RedShift and can read the data from redshift .
My requirement is to read the Resdhift data convert to CSV and which can be
later picked by Elastic Search .
Has anybody used NiFi for such usecase .
Appreciate if can explain how the data ingestion pipeline works .

Thanks,
Divya

Re: Data Ingestion pipeline - NiFi to ElasticSearch

[Mike Thomsen <mi...@gmail.com>]

It's done all the time. You'll want to use PutElasticsearchHttpRecord with
a CSVReader for the record reader. You won't need to write a schema for it
right away, but without one it will rely on schema inference which will
probably give you sub-optimal field generation (assuming you don't also
write an Elasticsearch schema). To write a schema on the data side for NiFi
to use for parsing the CSV into your preferred fields (names and types),
read up on Apache Avro. It's easy to get into and you can ask questions
here if you run into issues with how to map a CSV file to an avro schema in
NiFi.

On Mon, Aug 19, 2019 at 11:39 PM Divya Gehlot <di...@gmail.com>
wrote:

> Hi,
> I am working on Data Ingestion Pipeline  using NiFi   ingest data to
> Elastic Search.  Tech Stack :
> Apache NiFi - 1.9.2
> ElasticSearch 7.1.0
>
> I am able to connect to RedShift and can read the data from redshift .
> My requirement is to read the Resdhift data convert to CSV and which can
> be later picked by Elastic Search .
> Has anybody used NiFi for such usecase .
> Appreciate if can explain how the data ingestion pipeline works .
>
> Thanks,
> Divya
>