You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jagan <ja...@gmail.com> on 2016/09/22 19:01:53 UTC
Fwd: Flink Scala - data load from CSV to postgress database.
Hi Guys,
We have a requirement like – loading data from local CSV file to Postgress
database using Flink Scala…We have tried number of ways all failed
Do you have any example for this? With dependency libraries to understand
how to load data from CSV to postgres
We have tried and searched in Google/Flinkweb website for data load, we
haven’t found any sample code for this requisite.
Code: Flink Scala.
Load: from CSV to local postgres database.
Thanks
Jagan
0044-7411239688
--
Regards.
Jagan.
---------------------------------------------------------------------------------------------------------------------
***** The Good You Do Today, People will often forget tomorrow: Do Good
Anyway ****
Re: Flink Scala - data load from CSV to postgress database.
Posted by Jagan <ja...@gmail.com>.
Hi Flavio,
Thanks this script is very useful, however we need Flink scala script to
load data from csv to postgress.
if we have following java equivalent code in scala that will be really
helpful...thanks.
myResult.output(
// build and configure OutputFormat
JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername(" ")
.setDBUrl(" ")
.setQuery("insert into persons (name, age, height)
values (?,?,?)")
.finish()
);
On 23 September 2016 at 08:35, Flavio Pompermaier <po...@okkam.it>
wrote:
> Hi Jagan, I have a Java version of such a Job if you want (that requires
> table-api in order to handle null values), I hope this could help!
> Obviously you'll have to set all parameters properly..(e.g.
> jdbcOutputFormat ,
> fieldTypes, etc...).
>
>
> public static void main(String[] args) throws Exception {
> String path = "file:/tmp/myFile.csv";
> String rowDelim = CsvInputFormat.DEFAULT_LINE_DELIMITER;
> String fieldDelim = CsvInputFormat.DEFAULT_FIELD_DELIMITER;
> String[] fieldNames = "Column 1,Column 2,Column
> 3,Column4".split(fieldDelim);
>
> Character quoteCharacter = '"';
> boolean ignoreFirstLine = Boolean.TRUE;
> String ignoreComments = null;
> boolean lenient = Boolean.FALSE;
>
> TypeInformation<?>[] fieldTypes = new TypeInformation<?>[] {
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.INT_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO,
> BasicTypeInfo.STRING_TYPE_INFO
> };
>
> ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
>
> CsvTableSource csvTableSource = new
> CsvTableSource(path,fieldNames,fieldTypes,fieldDelim,
> rowDelim, quoteCharacter, ignoreFirstLine, ignoreComments, lenient);
> DataSet<Row> csvDataSet = csvTableSource.getDataSet(env);
> JDBCOutputFormat jdbcOutputFormat =
> JDBCOutputFormat.buildJDBCOutputFormat()
> .setDrivername("org.postgresql.Driver")
> .setDBUrl("jdbc:postgresql://localhost/test?user=xxx&password=xxx")
> .setQuery("insert into %s (id, title, author, price, qty) values
> (?,?,?,?,?)") .finish(); csvDataSet.output(jdbcOutputFormat);
> }
>
> Best,
> Flavio
>
> On Fri, Sep 23, 2016 at 12:52 AM, Jagan <ja...@gmail.com> wrote:
>
> > Hi Guys,
> >
> > We have requirement like – loading data from local CSV file to Postgress
> > database using Flink Scala…
> >
> > Do you have any sample Flink scala code for this?
> >
> > We have tried and searched in Google/Flinkweb website for data load, we
> > haven’t found any sample code for this requisite.
> >
> > Code: Flink Scala.
> >
> > Load: from CSV to local postgres database.
> >
> > Thanks
> >
> > Jagan
> >
> > On 22 September 2016 at 20:37, Jagan <ja...@gmail.com> wrote:
> >
> > > Hi Team,
> > >
> > > Will you be able to guide me on this? Is this a known issue that
> we
> > > can't implement dataload in flink scala ?
> > >
> > > data load from csv to postgress or any relational database in
> > Flink
> > > Scala
> > >
> > > Thanks
> > >
> > > Jagan.
> > >
> > > On 22 September 2016 at 20:15, Jagan <ja...@gmail.com> wrote:
> > >
> > >> Thanks Suneel,
> > >>
> > >> but client want to implement the data load in Flink Scala..
> > >>
> > >>
> > >> On 22 September 2016 at 20:07, Suneel Marthi <sm...@apache.org>
> > wrote:
> > >>
> > >>> Couldn't u use SQLLoader or something for doing that?
> > >>>
> > >>> http://stackoverflow.com/questions/2987433/how-to-import-csv
> > >>> -file-data-into-a-postgresql-table
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
> > >>>
> > >>> > Hi Guys,
> > >>> >
> > >>> > We have a requirement like – loading data from local CSV file to
> > >>> Postgress
> > >>> > database using Flink Scala…We have tried number of ways all failed
> > >>> >
> > >>> > Do you have any example for this? With dependency libraries to
> > >>> understand
> > >>> > how to load data from CSV to postgres
> > >>> >
> > >>> > We have tried and searched in Google/Flinkweb website for data
> load,
> > we
> > >>> > haven’t found any sample code for this requisite.
> > >>> >
> > >>> > Code: Flink Scala.
> > >>> >
> > >>> > Load: from CSV to local postgres database.
> > >>> >
> > >>> > Thanks
> > >>> >
> > >>> > Jagan
> > >>> >
> > >>> > 0044-7411239688
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Regards.
> > >>> > Jagan.
> > >>> >
> > >>> > ------------------------------------------------------------
> > >>> > ---------------------------------------------------------
> > >>> > ***** The Good You Do Today, People will often forget tomorrow: Do
> > >>> Good
> > >>> > Anyway ****
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Regards.
> > >> Jagan.
> > >>
> > >> ------------------------------------------------------------
> > >> ---------------------------------------------------------
> > >> ***** The Good You Do Today, People will often forget tomorrow: Do
> Good
> > >> Anyway ****
> > >>
> > >
> > >
> > >
> > > --
> > > Regards.
> > > Jagan.
> > >
> > > ------------------------------------------------------------
> > > ---------------------------------------------------------
> > > ***** The Good You Do Today, People will often forget tomorrow: Do
> Good
> > > Anyway ****
> > >
> >
> >
> >
> > --
> > Regards.
> > Jagan.
> >
> > ------------------------------------------------------------
> > ---------------------------------------------------------
> > ***** The Good You Do Today, People will often forget tomorrow: Do Good
> > Anyway ****
> >
>
--
Regards.
Jagan.
---------------------------------------------------------------------------------------------------------------------
***** The Good You Do Today, People will often forget tomorrow: Do Good
Anyway ****
Re: Flink Scala - data load from CSV to postgress database.
Posted by Flavio Pompermaier <po...@okkam.it>.
Hi Jagan, I have a Java version of such a Job if you want (that requires
table-api in order to handle null values), I hope this could help!
Obviously you'll have to set all parameters properly..(e.g. jdbcOutputFormat ,
fieldTypes, etc...).
public static void main(String[] args) throws Exception {
String path = "file:/tmp/myFile.csv";
String rowDelim = CsvInputFormat.DEFAULT_LINE_DELIMITER;
String fieldDelim = CsvInputFormat.DEFAULT_FIELD_DELIMITER;
String[] fieldNames = "Column 1,Column 2,Column
3,Column4".split(fieldDelim);
Character quoteCharacter = '"';
boolean ignoreFirstLine = Boolean.TRUE;
String ignoreComments = null;
boolean lenient = Boolean.FALSE;
TypeInformation<?>[] fieldTypes = new TypeInformation<?>[] {
BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.INT_TYPE_INFO,
BasicTypeInfo.STRING_TYPE_INFO,
BasicTypeInfo.STRING_TYPE_INFO
};
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
CsvTableSource csvTableSource = new
CsvTableSource(path,fieldNames,fieldTypes,fieldDelim,
rowDelim, quoteCharacter, ignoreFirstLine, ignoreComments, lenient);
DataSet<Row> csvDataSet = csvTableSource.getDataSet(env);
JDBCOutputFormat jdbcOutputFormat =
JDBCOutputFormat.buildJDBCOutputFormat()
.setDrivername("org.postgresql.Driver")
.setDBUrl("jdbc:postgresql://localhost/test?user=xxx&password=xxx")
.setQuery("insert into %s (id, title, author, price, qty) values
(?,?,?,?,?)") .finish(); csvDataSet.output(jdbcOutputFormat);
}
Best,
Flavio
On Fri, Sep 23, 2016 at 12:52 AM, Jagan <ja...@gmail.com> wrote:
> Hi Guys,
>
> We have requirement like – loading data from local CSV file to Postgress
> database using Flink Scala…
>
> Do you have any sample Flink scala code for this?
>
> We have tried and searched in Google/Flinkweb website for data load, we
> haven’t found any sample code for this requisite.
>
> Code: Flink Scala.
>
> Load: from CSV to local postgres database.
>
> Thanks
>
> Jagan
>
> On 22 September 2016 at 20:37, Jagan <ja...@gmail.com> wrote:
>
> > Hi Team,
> >
> > Will you be able to guide me on this? Is this a known issue that we
> > can't implement dataload in flink scala ?
> >
> > data load from csv to postgress or any relational database in
> Flink
> > Scala
> >
> > Thanks
> >
> > Jagan.
> >
> > On 22 September 2016 at 20:15, Jagan <ja...@gmail.com> wrote:
> >
> >> Thanks Suneel,
> >>
> >> but client want to implement the data load in Flink Scala..
> >>
> >>
> >> On 22 September 2016 at 20:07, Suneel Marthi <sm...@apache.org>
> wrote:
> >>
> >>> Couldn't u use SQLLoader or something for doing that?
> >>>
> >>> http://stackoverflow.com/questions/2987433/how-to-import-csv
> >>> -file-data-into-a-postgresql-table
> >>>
> >>>
> >>>
> >>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
> >>>
> >>> > Hi Guys,
> >>> >
> >>> > We have a requirement like – loading data from local CSV file to
> >>> Postgress
> >>> > database using Flink Scala…We have tried number of ways all failed
> >>> >
> >>> > Do you have any example for this? With dependency libraries to
> >>> understand
> >>> > how to load data from CSV to postgres
> >>> >
> >>> > We have tried and searched in Google/Flinkweb website for data load,
> we
> >>> > haven’t found any sample code for this requisite.
> >>> >
> >>> > Code: Flink Scala.
> >>> >
> >>> > Load: from CSV to local postgres database.
> >>> >
> >>> > Thanks
> >>> >
> >>> > Jagan
> >>> >
> >>> > 0044-7411239688
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Regards.
> >>> > Jagan.
> >>> >
> >>> > ------------------------------------------------------------
> >>> > ---------------------------------------------------------
> >>> > ***** The Good You Do Today, People will often forget tomorrow: Do
> >>> Good
> >>> > Anyway ****
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Regards.
> >> Jagan.
> >>
> >> ------------------------------------------------------------
> >> ---------------------------------------------------------
> >> ***** The Good You Do Today, People will often forget tomorrow: Do Good
> >> Anyway ****
> >>
> >
> >
> >
> > --
> > Regards.
> > Jagan.
> >
> > ------------------------------------------------------------
> > ---------------------------------------------------------
> > ***** The Good You Do Today, People will often forget tomorrow: Do Good
> > Anyway ****
> >
>
>
>
> --
> Regards.
> Jagan.
>
> ------------------------------------------------------------
> ---------------------------------------------------------
> ***** The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway ****
>
Re: Flink Scala - data load from CSV to postgress database.
Posted by Jagan <ja...@gmail.com>.
Hi Guys,
We have requirement like – loading data from local CSV file to Postgress
database using Flink Scala…
Do you have any sample Flink scala code for this?
We have tried and searched in Google/Flinkweb website for data load, we
haven’t found any sample code for this requisite.
Code: Flink Scala.
Load: from CSV to local postgres database.
Thanks
Jagan
On 22 September 2016 at 20:37, Jagan <ja...@gmail.com> wrote:
> Hi Team,
>
> Will you be able to guide me on this? Is this a known issue that we
> can't implement dataload in flink scala ?
>
> data load from csv to postgress or any relational database in Flink
> Scala
>
> Thanks
>
> Jagan.
>
> On 22 September 2016 at 20:15, Jagan <ja...@gmail.com> wrote:
>
>> Thanks Suneel,
>>
>> but client want to implement the data load in Flink Scala..
>>
>>
>> On 22 September 2016 at 20:07, Suneel Marthi <sm...@apache.org> wrote:
>>
>>> Couldn't u use SQLLoader or something for doing that?
>>>
>>> http://stackoverflow.com/questions/2987433/how-to-import-csv
>>> -file-data-into-a-postgresql-table
>>>
>>>
>>>
>>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > We have a requirement like – loading data from local CSV file to
>>> Postgress
>>> > database using Flink Scala…We have tried number of ways all failed
>>> >
>>> > Do you have any example for this? With dependency libraries to
>>> understand
>>> > how to load data from CSV to postgres
>>> >
>>> > We have tried and searched in Google/Flinkweb website for data load, we
>>> > haven’t found any sample code for this requisite.
>>> >
>>> > Code: Flink Scala.
>>> >
>>> > Load: from CSV to local postgres database.
>>> >
>>> > Thanks
>>> >
>>> > Jagan
>>> >
>>> > 0044-7411239688
>>> >
>>> >
>>> >
>>> > --
>>> > Regards.
>>> > Jagan.
>>> >
>>> > ------------------------------------------------------------
>>> > ---------------------------------------------------------
>>> > ***** The Good You Do Today, People will often forget tomorrow: Do
>>> Good
>>> > Anyway ****
>>> >
>>>
>>
>>
>>
>> --
>> Regards.
>> Jagan.
>>
>> ------------------------------------------------------------
>> ---------------------------------------------------------
>> ***** The Good You Do Today, People will often forget tomorrow: Do Good
>> Anyway ****
>>
>
>
>
> --
> Regards.
> Jagan.
>
> ------------------------------------------------------------
> ---------------------------------------------------------
> ***** The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway ****
>
--
Regards.
Jagan.
---------------------------------------------------------------------------------------------------------------------
***** The Good You Do Today, People will often forget tomorrow: Do Good
Anyway ****
Re: Flink Scala - data load from CSV to postgress database.
Posted by Jagan <ja...@gmail.com>.
Hi Team,
Will you be able to guide me on this? Is this a known issue that we
can't implement dataload in flink scala ?
data load from csv to postgress or any relational database in Flink
Scala
Thanks
Jagan.
On 22 September 2016 at 20:15, Jagan <ja...@gmail.com> wrote:
> Thanks Suneel,
>
> but client want to implement the data load in Flink Scala..
>
>
> On 22 September 2016 at 20:07, Suneel Marthi <sm...@apache.org> wrote:
>
>> Couldn't u use SQLLoader or something for doing that?
>>
>> http://stackoverflow.com/questions/2987433/how-to-import-
>> csv-file-data-into-a-postgresql-table
>>
>>
>>
>> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
>>
>> > Hi Guys,
>> >
>> > We have a requirement like – loading data from local CSV file to
>> Postgress
>> > database using Flink Scala…We have tried number of ways all failed
>> >
>> > Do you have any example for this? With dependency libraries to
>> understand
>> > how to load data from CSV to postgres
>> >
>> > We have tried and searched in Google/Flinkweb website for data load, we
>> > haven’t found any sample code for this requisite.
>> >
>> > Code: Flink Scala.
>> >
>> > Load: from CSV to local postgres database.
>> >
>> > Thanks
>> >
>> > Jagan
>> >
>> > 0044-7411239688
>> >
>> >
>> >
>> > --
>> > Regards.
>> > Jagan.
>> >
>> > ------------------------------------------------------------
>> > ---------------------------------------------------------
>> > ***** The Good You Do Today, People will often forget tomorrow: Do Good
>> > Anyway ****
>> >
>>
>
>
>
> --
> Regards.
> Jagan.
>
> ------------------------------------------------------------
> ---------------------------------------------------------
> ***** The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway ****
>
--
Regards.
Jagan.
---------------------------------------------------------------------------------------------------------------------
***** The Good You Do Today, People will often forget tomorrow: Do Good
Anyway ****
Re: Flink Scala - data load from CSV to postgress database.
Posted by Jagan <ja...@gmail.com>.
Thanks Suneel,
but client want to implement the data load in Flink Scala..
On 22 September 2016 at 20:07, Suneel Marthi <sm...@apache.org> wrote:
> Couldn't u use SQLLoader or something for doing that?
>
> http://stackoverflow.com/questions/2987433/how-to-
> import-csv-file-data-into-a-postgresql-table
>
>
>
> On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
>
> > Hi Guys,
> >
> > We have a requirement like – loading data from local CSV file to
> Postgress
> > database using Flink Scala…We have tried number of ways all failed
> >
> > Do you have any example for this? With dependency libraries to understand
> > how to load data from CSV to postgres
> >
> > We have tried and searched in Google/Flinkweb website for data load, we
> > haven’t found any sample code for this requisite.
> >
> > Code: Flink Scala.
> >
> > Load: from CSV to local postgres database.
> >
> > Thanks
> >
> > Jagan
> >
> > 0044-7411239688
> >
> >
> >
> > --
> > Regards.
> > Jagan.
> >
> > ------------------------------------------------------------
> > ---------------------------------------------------------
> > ***** The Good You Do Today, People will often forget tomorrow: Do Good
> > Anyway ****
> >
>
--
Regards.
Jagan.
---------------------------------------------------------------------------------------------------------------------
***** The Good You Do Today, People will often forget tomorrow: Do Good
Anyway ****
Re: Flink Scala - data load from CSV to postgress database.
Posted by Suneel Marthi <sm...@apache.org>.
Couldn't u use SQLLoader or something for doing that?
http://stackoverflow.com/questions/2987433/how-to-import-csv-file-data-into-a-postgresql-table
On Thu, Sep 22, 2016 at 3:01 PM, Jagan <ja...@gmail.com> wrote:
> Hi Guys,
>
> We have a requirement like – loading data from local CSV file to Postgress
> database using Flink Scala…We have tried number of ways all failed
>
> Do you have any example for this? With dependency libraries to understand
> how to load data from CSV to postgres
>
> We have tried and searched in Google/Flinkweb website for data load, we
> haven’t found any sample code for this requisite.
>
> Code: Flink Scala.
>
> Load: from CSV to local postgres database.
>
> Thanks
>
> Jagan
>
> 0044-7411239688
>
>
>
> --
> Regards.
> Jagan.
>
> ------------------------------------------------------------
> ---------------------------------------------------------
> ***** The Good You Do Today, People will often forget tomorrow: Do Good
> Anyway ****
>