You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Jaikit Jilka <jj...@leadferret.com> on 2016/07/14 22:53:10 UTC
Operators not working
Hello,
I am parsing data from one table to another table. My operators are running but it does not parse or emit anything after it parses 50000 rows. I also partitioned my parser operator and jdbcoutput operator. It 50000 it works fine and later it slows down and finally it does not emit anything. Latency of the operator is also around 120 - 150. What can be wrong here.
Thank You,
Jaikit JIlka
Re: Operators not working
Posted by Ashwin Chandra Putta <as...@gmail.com>.
I remember the jdbc pojo input operator did only batch read (only existing
records at the time of start of the operator) and not continuous polling of
subsequent records. Not sure if this is fixed?
Can you try the polling input operator?
https://github.com/devtagare/examples/tree/master/tutorials/JdbcIngest
Regards,
Ashwin.
On Thu, Jul 21, 2016 at 11:11 AM, Pradeep A. Dalvi <pr...@apache.org> wrote:
> What happens if you remove JDBC Output Operator and write to console or
> something, do you still observe the similar behavior?
>
> --prad
>
> On Thu, Jul 21, 2016 at 10:58 AM, Jaikit Jilka <jj...@leadferret.com>
> wrote:
>
>> Hello,
>>
>> I am referring to JdbcPojoImput operator. when I launch the application
>> it emits around 500records/sec but once it reaches 30k it slows down and
>> emits 15records/sec. I inserting record from a table which has data coming
>> in continously and after the data is parsed I am deleting the record from
>> source table. But this is not the issue because I tried removing delete
>> query and still gives me same error.
>>
>> ----- Original Message -----
>> From: "Ashwin Chandra Putta" <as...@gmail.com>
>> To: "users" <us...@apex.apache.org>
>> Sent: Thursday, July 14, 2016 6:41:18 PM
>> Subject: Re: Operators not working
>>
>> When you say it does not emit anything, are you referring to your input
>> operator? Is it an operator from malhar? Which one?
>>
>> And how many records do you have in the source table?
>>
>> Regards,
>> Ashwin.
>>
>> On Jul 14, 2016 3:55 PM, "Jaikit Jilka" <jj...@leadferret.com> wrote:
>>
>> > Hello,
>> >
>> > I am parsing data from one table to another table. My operators are
>> > running but it does not parse or emit anything after it parses 50000
>> rows.
>> > I also partitioned my parser operator and jdbcoutput operator. It 50000
>> it
>> > works fine and later it slows down and finally it does not emit
>> anything.
>> > Latency of the operator is also around 120 - 150. What can be wrong
>> here.
>> >
>> > Thank You,
>> > Jaikit JIlka
>> >
>>
>
>
--
Regards,
Ashwin.
Re: Operators not working
Posted by Jaikit Jilka <jj...@leadferret.com>.
Hello,
I replaced my JdbcPojoOutput operator with console operator and still it behaves the same. It started with 600record/sec and now it is running at 30records/sec.
----- Original Message -----
From: "Pradeep A. Dalvi" <pr...@apache.org>
To: "users" <us...@apex.apache.org>
Sent: Thursday, July 21, 2016 11:11:17 AM
Subject: Re: Operators not working
What happens if you remove JDBC Output Operator and write to console or
something, do you still observe the similar behavior?
--prad
On Thu, Jul 21, 2016 at 10:58 AM, Jaikit Jilka <jj...@leadferret.com>
wrote:
> Hello,
>
> I am referring to JdbcPojoImput operator. when I launch the application it
> emits around 500records/sec but once it reaches 30k it slows down and emits
> 15records/sec. I inserting record from a table which has data coming in
> continously and after the data is parsed I am deleting the record from
> source table. But this is not the issue because I tried removing delete
> query and still gives me same error.
>
> ----- Original Message -----
> From: "Ashwin Chandra Putta" <as...@gmail.com>
> To: "users" <us...@apex.apache.org>
> Sent: Thursday, July 14, 2016 6:41:18 PM
> Subject: Re: Operators not working
>
> When you say it does not emit anything, are you referring to your input
> operator? Is it an operator from malhar? Which one?
>
> And how many records do you have in the source table?
>
> Regards,
> Ashwin.
>
> On Jul 14, 2016 3:55 PM, "Jaikit Jilka" <jj...@leadferret.com> wrote:
>
> > Hello,
> >
> > I am parsing data from one table to another table. My operators are
> > running but it does not parse or emit anything after it parses 50000
> rows.
> > I also partitioned my parser operator and jdbcoutput operator. It 50000
> it
> > works fine and later it slows down and finally it does not emit anything.
> > Latency of the operator is also around 120 - 150. What can be wrong here.
> >
> > Thank You,
> > Jaikit JIlka
> >
>
Re: Operators not working
Posted by "Pradeep A. Dalvi" <pr...@apache.org>.
What happens if you remove JDBC Output Operator and write to console or
something, do you still observe the similar behavior?
--prad
On Thu, Jul 21, 2016 at 10:58 AM, Jaikit Jilka <jj...@leadferret.com>
wrote:
> Hello,
>
> I am referring to JdbcPojoImput operator. when I launch the application it
> emits around 500records/sec but once it reaches 30k it slows down and emits
> 15records/sec. I inserting record from a table which has data coming in
> continously and after the data is parsed I am deleting the record from
> source table. But this is not the issue because I tried removing delete
> query and still gives me same error.
>
> ----- Original Message -----
> From: "Ashwin Chandra Putta" <as...@gmail.com>
> To: "users" <us...@apex.apache.org>
> Sent: Thursday, July 14, 2016 6:41:18 PM
> Subject: Re: Operators not working
>
> When you say it does not emit anything, are you referring to your input
> operator? Is it an operator from malhar? Which one?
>
> And how many records do you have in the source table?
>
> Regards,
> Ashwin.
>
> On Jul 14, 2016 3:55 PM, "Jaikit Jilka" <jj...@leadferret.com> wrote:
>
> > Hello,
> >
> > I am parsing data from one table to another table. My operators are
> > running but it does not parse or emit anything after it parses 50000
> rows.
> > I also partitioned my parser operator and jdbcoutput operator. It 50000
> it
> > works fine and later it slows down and finally it does not emit anything.
> > Latency of the operator is also around 120 - 150. What can be wrong here.
> >
> > Thank You,
> > Jaikit JIlka
> >
>
Re: Operators not working
Posted by Jaikit Jilka <jj...@leadferret.com>.
Hello,
I am referring to JdbcPojoImput operator. when I launch the application it emits around 500records/sec but once it reaches 30k it slows down and emits 15records/sec. I inserting record from a table which has data coming in continously and after the data is parsed I am deleting the record from source table. But this is not the issue because I tried removing delete query and still gives me same error.
----- Original Message -----
From: "Ashwin Chandra Putta" <as...@gmail.com>
To: "users" <us...@apex.apache.org>
Sent: Thursday, July 14, 2016 6:41:18 PM
Subject: Re: Operators not working
When you say it does not emit anything, are you referring to your input
operator? Is it an operator from malhar? Which one?
And how many records do you have in the source table?
Regards,
Ashwin.
On Jul 14, 2016 3:55 PM, "Jaikit Jilka" <jj...@leadferret.com> wrote:
> Hello,
>
> I am parsing data from one table to another table. My operators are
> running but it does not parse or emit anything after it parses 50000 rows.
> I also partitioned my parser operator and jdbcoutput operator. It 50000 it
> works fine and later it slows down and finally it does not emit anything.
> Latency of the operator is also around 120 - 150. What can be wrong here.
>
> Thank You,
> Jaikit JIlka
>
Re: Operators not working
Posted by Ashwin Chandra Putta <as...@gmail.com>.
When you say it does not emit anything, are you referring to your input
operator? Is it an operator from malhar? Which one?
And how many records do you have in the source table?
Regards,
Ashwin.
On Jul 14, 2016 3:55 PM, "Jaikit Jilka" <jj...@leadferret.com> wrote:
> Hello,
>
> I am parsing data from one table to another table. My operators are
> running but it does not parse or emit anything after it parses 50000 rows.
> I also partitioned my parser operator and jdbcoutput operator. It 50000 it
> works fine and later it slows down and finally it does not emit anything.
> Latency of the operator is also around 120 - 150. What can be wrong here.
>
> Thank You,
> Jaikit JIlka
>