You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apisix.apache.org by "~Jarvis.Qiu " <qi...@foxmail.com> on 2020/03/31 10:00:21 UTC

[DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

the code in: https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101

The limit of lua_max_pending_timers and lua_max_running_timers that may be reached.

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Ming Wen <mo...@gmail.com>.
+1. I think `batch processor` is good for me.

Thanks,
Ming Wen
Twitter: _WenMing


YuanSheng Wang <me...@apache.org> 于2020年3月31日周二 下午10:05写道：

>
>
> On Tue, Mar 31, 2020 at 8:07 PM Ayeshmantha Perera <
> akayeshmantha@gmail.com> wrote:
>
>> @YuanSheng Wang <me...@apache.org>  @Ming Wen <mo...@gmail.com>
>> What do you think shall we remove the Async support of the Kafka library
>> and move forward with the internal buffering
>>
>
> agree +1
>
> ^_^
>
>
>>
>> On Tue, Mar 31, 2020 at 2:05 PM Ayeshmantha Perera <
>> akayeshmantha@gmail.com> wrote:
>>
>>> Yeah that is a good idea
>>>
>>> On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I created a PR to use the batch processor which executes only one-timer
>>>> between the `buffer_duration`. I guess this can be used to resolve the
>>>> problem.
>>>>
>>>> https://github.com/apache/incubator-apisix/pull/1358
>>>>
>>>>
>>>> On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
>>>> wrote:
>>>>
>>>> > I think the code is used to work around the 'cosocket API disabled'
>>>> > limitation in log phase.
>>>> > I agree with you that we should take care about the max of the timers.
>>>> > Perhaps, we can buffer the data and use constant number of timer to
>>>> flush
>>>> > it?
>>>> >
>>>> > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
>>>> >
>>>> > > the code in:
>>>> > >
>>>> >
>>>> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
>>>> > >
>>>> > > The limit of lua_max_pending_timers and lua_max_running_timers that
>>>> may
>>>> > be
>>>> > > reached.
>>>> >
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> S.Nirojan
>>>>
>>>> Research Engineer
>>>> Linked-in: https://www.linkedin.com/in/niroselva/
>>>> Mobile : (+94) 779823445 / 067763149095
>>>>
>>>
>>>
>>> --
>>> *Software Engineer*
>>> *Salzburg Research Forschungsgesellschaft *
>>> *Salzburg, Austria*
>>>
>>
>>
>> --
>> *Software Engineer*
>> *Salzburg Research Forschungsgesellschaft *
>> *Salzburg, Austria*
>>
>
>
> --
> *MembPhis*
> My github: https://github.com/membphis
> Apache APISIX: https://github.com/apache/incubator-apisix
>

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by YuanSheng Wang <me...@apache.org>.
On Tue, Mar 31, 2020 at 8:07 PM Ayeshmantha Perera <ak...@gmail.com>
wrote:

> @YuanSheng Wang <me...@apache.org>  @Ming Wen <mo...@gmail.com>
> What do you think shall we remove the Async support of the Kafka library
> and move forward with the internal buffering
>

agree +1

^_^


>
> On Tue, Mar 31, 2020 at 2:05 PM Ayeshmantha Perera <
> akayeshmantha@gmail.com> wrote:
>
>> Yeah that is a good idea
>>
>> On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I created a PR to use the batch processor which executes only one-timer
>>> between the `buffer_duration`. I guess this can be used to resolve the
>>> problem.
>>>
>>> https://github.com/apache/incubator-apisix/pull/1358
>>>
>>>
>>> On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
>>> wrote:
>>>
>>> > I think the code is used to work around the 'cosocket API disabled'
>>> > limitation in log phase.
>>> > I agree with you that we should take care about the max of the timers.
>>> > Perhaps, we can buffer the data and use constant number of timer to
>>> flush
>>> > it?
>>> >
>>> > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
>>> >
>>> > > the code in:
>>> > >
>>> >
>>> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
>>> > >
>>> > > The limit of lua_max_pending_timers and lua_max_running_timers that
>>> may
>>> > be
>>> > > reached.
>>> >
>>>
>>>
>>> --
>>> Best Regards,
>>> S.Nirojan
>>>
>>> Research Engineer
>>> Linked-in: https://www.linkedin.com/in/niroselva/
>>> Mobile : (+94) 779823445 / 067763149095
>>>
>>
>>
>> --
>> *Software Engineer*
>> *Salzburg Research Forschungsgesellschaft *
>> *Salzburg, Austria*
>>
>
>
> --
> *Software Engineer*
> *Salzburg Research Forschungsgesellschaft *
> *Salzburg, Austria*
>


-- 
*MembPhis*
My github: https://github.com/membphis
Apache APISIX: https://github.com/apache/incubator-apisix

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Ayeshmantha Perera <ak...@gmail.com>.
@YuanSheng Wang <me...@apache.org>  @Ming Wen <mo...@gmail.com>
What do you think shall we remove the Async support of the Kafka library
and move forward with the internal buffering

On Tue, Mar 31, 2020 at 2:05 PM Ayeshmantha Perera <ak...@gmail.com>
wrote:

> Yeah that is a good idea
>
> On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I created a PR to use the batch processor which executes only one-timer
>> between the `buffer_duration`. I guess this can be used to resolve the
>> problem.
>>
>> https://github.com/apache/incubator-apisix/pull/1358
>>
>>
>> On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
>> wrote:
>>
>> > I think the code is used to work around the 'cosocket API disabled'
>> > limitation in log phase.
>> > I agree with you that we should take care about the max of the timers.
>> > Perhaps, we can buffer the data and use constant number of timer to
>> flush
>> > it?
>> >
>> > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
>> >
>> > > the code in:
>> > >
>> >
>> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
>> > >
>> > > The limit of lua_max_pending_timers and lua_max_running_timers that
>> may
>> > be
>> > > reached.
>> >
>>
>>
>> --
>> Best Regards,
>> S.Nirojan
>>
>> Research Engineer
>> Linked-in: https://www.linkedin.com/in/niroselva/
>> Mobile : (+94) 779823445 / 067763149095
>>
>
>
> --
> *Software Engineer*
> *Salzburg Research Forschungsgesellschaft *
> *Salzburg, Austria*
>


-- 
*Software Engineer*
*Salzburg Research Forschungsgesellschaft *
*Salzburg, Austria*

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Ayeshmantha Perera <ak...@gmail.com>.
Yeah that is a good idea

On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
wrote:

> Hi,
>
> I created a PR to use the batch processor which executes only one-timer
> between the `buffer_duration`. I guess this can be used to resolve the
> problem.
>
> https://github.com/apache/incubator-apisix/pull/1358
>
>
> On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
> wrote:
>
> > I think the code is used to work around the 'cosocket API disabled'
> > limitation in log phase.
> > I agree with you that we should take care about the max of the timers.
> > Perhaps, we can buffer the data and use constant number of timer to flush
> > it?
> >
> > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
> >
> > > the code in:
> > >
> >
> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
> > >
> > > The limit of lua_max_pending_timers and lua_max_running_timers that may
> > be
> > > reached.
> >
>
>
> --
> Best Regards,
> S.Nirojan
>
> Research Engineer
> Linked-in: https://www.linkedin.com/in/niroselva/
> Mobile : (+94) 779823445 / 067763149095
>


-- 
*Software Engineer*
*Salzburg Research Forschungsgesellschaft *
*Salzburg, Austria*

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Nirojan Selvanathan <ss...@gmail.com>.
we can use the internal APISX buffering instead of the async method I
presume.

On Tue, Mar 31, 2020 at 1:42 PM Ayeshmantha Perera <ak...@gmail.com>
wrote:

> Yeah, I have to agree with you all on this.
>
> But is there any way we can just execute the function call without the
> timer.at(). because if we are doing buffering from our end for async calls
> there will be two points this buffering happens from our batch processor as
> well as the library inbuilt batch processor both. If not then I think the
> best way is to use the Batch processor from apisix as Nirojan does.
>
> Regards
> Ayesh
>
> On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I created a PR to use the batch processor which executes only one-timer
> > between the `buffer_duration`. I guess this can be used to resolve the
> > problem.
> >
> > https://github.com/apache/incubator-apisix/pull/1358
> >
> >
> > On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
> > wrote:
> >
> > > I think the code is used to work around the 'cosocket API disabled'
> > > limitation in log phase.
> > > I agree with you that we should take care about the max of the timers.
> > > Perhaps, we can buffer the data and use constant number of timer to
> flush
> > > it?
> > >
> > > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
> > >
> > > > the code in:
> > > >
> > >
> >
> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
> > > >
> > > > The limit of lua_max_pending_timers and lua_max_running_timers that
> may
> > > be
> > > > reached.
> > >
> >
> >
> > --
> > Best Regards,
> > S.Nirojan
> >
> > Research Engineer
> > Linked-in: https://www.linkedin.com/in/niroselva/
> > Mobile : (+94) 779823445 / 067763149095
> >
>
>
> --
> *Software Engineer*
> *Salzburg Research Forschungsgesellschaft *
> *Salzburg, Austria*
>


-- 
Best Regards,
S.Nirojan

Research Engineer
Linked-in: https://www.linkedin.com/in/niroselva/
Mobile : (+94) 779823445 / 067763149095

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Ayeshmantha Perera <ak...@gmail.com>.
Yeah, I have to agree with you all on this.

But is there any way we can just execute the function call without the
timer.at(). because if we are doing buffering from our end for async calls
there will be two points this buffering happens from our batch processor as
well as the library inbuilt batch processor both. If not then I think the
best way is to use the Batch processor from apisix as Nirojan does.

Regards
Ayesh

On Tue, Mar 31, 2020 at 12:59 PM Nirojan Selvanathan <ss...@gmail.com>
wrote:

> Hi,
>
> I created a PR to use the batch processor which executes only one-timer
> between the `buffer_duration`. I guess this can be used to resolve the
> problem.
>
> https://github.com/apache/incubator-apisix/pull/1358
>
>
> On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
> wrote:
>
> > I think the code is used to work around the 'cosocket API disabled'
> > limitation in log phase.
> > I agree with you that we should take care about the max of the timers.
> > Perhaps, we can buffer the data and use constant number of timer to flush
> > it?
> >
> > ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
> >
> > > the code in:
> > >
> >
> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
> > >
> > > The limit of lua_max_pending_timers and lua_max_running_timers that may
> > be
> > > reached.
> >
>
>
> --
> Best Regards,
> S.Nirojan
>
> Research Engineer
> Linked-in: https://www.linkedin.com/in/niroselva/
> Mobile : (+94) 779823445 / 067763149095
>


-- 
*Software Engineer*
*Salzburg Research Forschungsgesellschaft *
*Salzburg, Austria*

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Nirojan Selvanathan <ss...@gmail.com>.
Hi,

I created a PR to use the batch processor which executes only one-timer
between the `buffer_duration`. I guess this can be used to resolve the
problem.

https://github.com/apache/incubator-apisix/pull/1358


On Tue, Mar 31, 2020 at 12:10 PM Zexuan Luo <sp...@gmail.com>
wrote:

> I think the code is used to work around the 'cosocket API disabled'
> limitation in log phase.
> I agree with you that we should take care about the max of the timers.
> Perhaps, we can buffer the data and use constant number of timer to flush
> it?
>
> ~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：
>
> > the code in:
> >
> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
> >
> > The limit of lua_max_pending_timers and lua_max_running_timers that may
> be
> > reached.
>


-- 
Best Regards,
S.Nirojan

Research Engineer
Linked-in: https://www.linkedin.com/in/niroselva/
Mobile : (+94) 779823445 / 067763149095

Re: [DISCUSS] using kafka asynchronous producer is better than timer.at for kafka-logger plugin?

Posted by Zexuan Luo <sp...@gmail.com>.
I think the code is used to work around the 'cosocket API disabled'
limitation in log phase.
I agree with you that we should take care about the max of the timers.
Perhaps, we can buffer the data and use constant number of timer to flush
it?

~Jarvis.Qiu <qi...@foxmail.com> 于2020年3月31日周二 下午6:00写道：

> the code in:
> https://github.com/apache/incubator-apisix/blob/master/lua/apisix/plugins/kafka-logger.lua#L101
>
> The limit of lua_max_pending_timers and lua_max_running_timers that may be
> reached.