RE: How to deal with dynamic types

["Newport, Billy" <Bi...@gs.com>]

We’ve been using genericRecords with custom serializers to do exactly this. We need to run the same flink pipeline for 10s of thousands of different schemas for our use cases and code gening or building that many different jars just isn’t practical.


From: madan [mailto:madan.yellanki@gmail.com]
Sent: Monday, December 11, 2017 10:13 AM
To: user@flink.apache.org
Subject: How to deal with dynamic types

Hi,

I am trying some initial samples with flink. I have one doubt regarding data types. Flink support data types Tuple(max 25 fields), Java POJOs, Primitive types, Regular classes etc.,
In my case I do not have fixed type. I have meta data with filed names & its types. For ex., (Id:int, Name:String, Salary:Double, Dept:String,... etc). I do not know the number of fields, its names or types till I receive metadata. In these what should be the source type I should go with? Please suggest. Small example would be of great help.


Scenario trying to solve :

Input :
            Metadata : {"id":"int", "Name":"String","Salary":"Double","Dept":"String"}
            Data file :   csv data file with above fields data

Output required is : Calculate average of salary by department wise.


--
Thank you,
Madan.