You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by rey26 <re...@gmail.com> on 2018/08/17 19:02:56 UTC
PutSql Loading is slow with Support Fragmented Transactions as
True...
I have a flow where i am putting data into Teradata database from csv files.I
am reading with csv and using convert record processor with source as
CSVReader and Target as JSON Reader and than JSONtoSql and then Putsql, but
loading is quite slow.
Since in the csv each file , each line is independent of each other , i
though to mark Support Fragmented Transactions as False and it improved the
load performance.
Can someone explains why its default value is True ?
Does it will have any impact on data?
Whats the exact use of this parameter?
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: PutSql Loading is slow with Support Fragmented Transactions as
True...
Posted by Mark Payne <ma...@hotmail.com>.
Hello,
That property will basically determine whether pushing the data is transactional or not.
For example, if you want to ensure that all data from your CSV is pushed or none of it is,
then you'll want that to be true. If you're okay with pushing the data in a bit at a time, then
you can set it to false.
Now, that said, the flow that you're using to convert CSV to JSON, then JSON to SQL, then
put SQL is fairly laborious and slow. Instead, I would recommend replacing all 3 ConvertRecord,
ConvertJSONToSql, and PutSQL Processor with a single PutDatabaseRecord processor. Then
you can put the CSV data directly to the database without the need to convert to JSON or SQL,
and this should be faster.
Thanks
-Mark
> On Aug 17, 2018, at 3:02 PM, rey26 <re...@gmail.com> wrote:
>
> I have a flow where i am putting data into Teradata database from csv files.I
> am reading with csv and using convert record processor with source as
> CSVReader and Target as JSON Reader and than JSONtoSql and then Putsql, but
> loading is quite slow.
>
> Since in the csv each file , each line is independent of each other , i
> though to mark Support Fragmented Transactions as False and it improved the
> load performance.
>
> Can someone explains why its default value is True ?
> Does it will have any impact on data?
> Whats the exact use of this parameter?
>
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/