You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by dafang <13...@163.com> on 2021/09/11 01:46:37 UTC
A question about skywalking-kong
Hi skywalking dev team:
I had read the source code in skywalking-kong,and I found that the request_uri is wrote in request case.But I have heard that if the first case if not set the url templated,the after will never set it again.If it's set up,our endpoint will be double or triple or more.It will be a very big problem for me.How to understand it?
英文不太好，怕表达不清，再用中文叙述一下：
我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
yours
大方
2021.09.11
Re: A question about skywalking-kong
Posted by Sheng Wu <wu...@gmail.com>.
Once you format them, it is fine.
dafang <13...@163.com>于2021年9月11日 周六下午5:12写道：
>
>
>
>
>
>
>
>
>
> I got what you mean. The fact that the problem was an trouble. And I now
> use a async process to transfer uri template. There is no other way.
>
>
>
>
>
>
>
>
> 在 2021-09-11 13:57:15，"Sheng Wu" <wu...@gmail.com> 写道：
> >And, if you don't set up alarm rule based on endpoint name, SkyWalking
> >wouldn't face OOM risk.
> >We are processing data in streaming mode, it makes resource costs
> >increased, but still limited.
> >
> >dafang <13...@163.com>于2021年9月11日 周六下午1:31写道：
> >
> >>
> >>
> >>
> >> Yeap,in our product design,some url is contains user id ,such as
> >> /hello/11230（/hello/{userId}）,and our user number is more than
> 10000000,in
> >> your example,it makes a lot trouble.How to deal this problem in any
> system
> >> in front of controller?
> >>
> >>
> >> 附：那所有在java controller之上的系统都会有这个问题吧，这个该怎么解决呢？
> >>
> >>
> 然后我还发现，对于span来说，我说的这些url是作为endpointName来存储和展示的，url中的动态参数如果不模板化替换，任何场景都会造成内存溢出问题
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> 在 2021-09-11 10:51:36，"Sheng Wu" <wu...@gmail.com> 写道：
> >> >This is not about Kong, it is for all services, even same for service
> >> mesh.
> >> >This thing is very simple, URI is the entity ID to measure metrics. If
> you
> >> >put parameter in the URI, such as /prod/123, then this is an entity.
> When
> >> >you have 1million product IDs, then you face potential OOM easily.
> >> >
> >> >dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
> >> >
> >> >> Hi skywalking dev team:
> >> >> I had read the source code in skywalking-kong,and I found that the
> >> >> request_uri is wrote in request case.But I have heard that if the
> first
> >> >> case if not set the url templated,the after will never set it
> again.If
> >> it's
> >> >> set up,our endpoint will be double or triple or more.It will be a
> very
> >> big
> >> >> problem for me.How to understand it?
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 英文不太好，怕表达不清，再用中文叙述一下：
> >> >>
> >> >>
> >>
> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
> >> >>
> >> >>
> >> >> yours
> >> >> 大方
> >> >> 2021.09.11
> >> >
> >> >--
> >> >Sheng Wu 吴晟
> >> >
> >> >Apache SkyWalking
> >> >Apache Incubator
> >> >Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >> >Zipkin
> >> >Twitter, wusheng1108
> >>
> >--
> >Sheng Wu 吴晟
> >
> >Apache SkyWalking
> >Apache Incubator
> >Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >Zipkin
> >Twitter, wusheng1108
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: A question about skywalking-kong
Posted by dafang <13...@163.com>.
I got what you mean. The fact that the problem was an trouble. And I now use a async process to transfer uri template. There is no other way.
在 2021-09-11 13:57:15，"Sheng Wu" <wu...@gmail.com> 写道：
>And, if you don't set up alarm rule based on endpoint name, SkyWalking
>wouldn't face OOM risk.
>We are processing data in streaming mode, it makes resource costs
>increased, but still limited.
>
>dafang <13...@163.com>于2021年9月11日 周六下午1:31写道：
>
>>
>>
>>
>> Yeap,in our product design,some url is contains user id ,such as
>> /hello/11230（/hello/{userId}）,and our user number is more than 10000000,in
>> your example,it makes a lot trouble.How to deal this problem in any system
>> in front of controller?
>>
>>
>> 附：那所有在java controller之上的系统都会有这个问题吧，这个该怎么解决呢？
>>
>> 然后我还发现，对于span来说，我说的这些url是作为endpointName来存储和展示的，url中的动态参数如果不模板化替换，任何场景都会造成内存溢出问题
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 在 2021-09-11 10:51:36，"Sheng Wu" <wu...@gmail.com> 写道：
>> >This is not about Kong, it is for all services, even same for service
>> mesh.
>> >This thing is very simple, URI is the entity ID to measure metrics. If you
>> >put parameter in the URI, such as /prod/123, then this is an entity. When
>> >you have 1million product IDs, then you face potential OOM easily.
>> >
>> >dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
>> >
>> >> Hi skywalking dev team:
>> >> I had read the source code in skywalking-kong,and I found that the
>> >> request_uri is wrote in request case.But I have heard that if the first
>> >> case if not set the url templated,the after will never set it again.If
>> it's
>> >> set up,our endpoint will be double or triple or more.It will be a very
>> big
>> >> problem for me.How to understand it?
>> >>
>> >>
>> >>
>> >>
>> >> 英文不太好，怕表达不清，再用中文叙述一下：
>> >>
>> >>
>> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
>> >>
>> >>
>> >> yours
>> >> 大方
>> >> 2021.09.11
>> >
>> >--
>> >Sheng Wu 吴晟
>> >
>> >Apache SkyWalking
>> >Apache Incubator
>> >Apache ShardingSphere, ECharts, DolphinScheduler podlings
>> >Zipkin
>> >Twitter, wusheng1108
>>
>--
>Sheng Wu 吴晟
>
>Apache SkyWalking
>Apache Incubator
>Apache ShardingSphere, ECharts, DolphinScheduler podlings
>Zipkin
>Twitter, wusheng1108
Re: A question about skywalking-kong
Posted by Sheng Wu <wu...@gmail.com>.
And, if you don't set up alarm rule based on endpoint name, SkyWalking
wouldn't face OOM risk.
We are processing data in streaming mode, it makes resource costs
increased, but still limited.
dafang <13...@163.com>于2021年9月11日 周六下午1:31写道：
>
>
>
> Yeap,in our product design,some url is contains user id ,such as
> /hello/11230（/hello/{userId}）,and our user number is more than 10000000,in
> your example,it makes a lot trouble.How to deal this problem in any system
> in front of controller?
>
>
> 附：那所有在java controller之上的系统都会有这个问题吧，这个该怎么解决呢？
>
> 然后我还发现，对于span来说，我说的这些url是作为endpointName来存储和展示的，url中的动态参数如果不模板化替换，任何场景都会造成内存溢出问题
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 在 2021-09-11 10:51:36，"Sheng Wu" <wu...@gmail.com> 写道：
> >This is not about Kong, it is for all services, even same for service
> mesh.
> >This thing is very simple, URI is the entity ID to measure metrics. If you
> >put parameter in the URI, such as /prod/123, then this is an entity. When
> >you have 1million product IDs, then you face potential OOM easily.
> >
> >dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
> >
> >> Hi skywalking dev team:
> >> I had read the source code in skywalking-kong,and I found that the
> >> request_uri is wrote in request case.But I have heard that if the first
> >> case if not set the url templated,the after will never set it again.If
> it's
> >> set up,our endpoint will be double or triple or more.It will be a very
> big
> >> problem for me.How to understand it?
> >>
> >>
> >>
> >>
> >> 英文不太好，怕表达不清，再用中文叙述一下：
> >>
> >>
> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
> >>
> >>
> >> yours
> >> 大方
> >> 2021.09.11
> >
> >--
> >Sheng Wu 吴晟
> >
> >Apache SkyWalking
> >Apache Incubator
> >Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >Zipkin
> >Twitter, wusheng1108
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: A question about skywalking-kong
Posted by Sheng Wu <wu...@gmail.com>.
Java agent could process this easily, we just build plugin for SpringMVC
control, then we can read the expression for general path.
At the backend, there are ways, regex or OpenAPI, to format paths. It just
costs backend resources.
dafang <13...@163.com>于2021年9月11日 周六下午1:31写道：
>
>
>
> Yeap,in our product design,some url is contains user id ,such as
> /hello/11230（/hello/{userId}）,and our user number is more than 10000000,in
> your example,it makes a lot trouble.How to deal this problem in any system
> in front of controller?
>
>
> 附：那所有在java controller之上的系统都会有这个问题吧，这个该怎么解决呢？
>
> 然后我还发现，对于span来说，我说的这些url是作为endpointName来存储和展示的，url中的动态参数如果不模板化替换，任何场景都会造成内存溢出问题
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 在 2021-09-11 10:51:36，"Sheng Wu" <wu...@gmail.com> 写道：
> >This is not about Kong, it is for all services, even same for service
> mesh.
> >This thing is very simple, URI is the entity ID to measure metrics. If you
> >put parameter in the URI, such as /prod/123, then this is an entity. When
> >you have 1million product IDs, then you face potential OOM easily.
> >
> >dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
> >
> >> Hi skywalking dev team:
> >> I had read the source code in skywalking-kong,and I found that the
> >> request_uri is wrote in request case.But I have heard that if the first
> >> case if not set the url templated,the after will never set it again.If
> it's
> >> set up,our endpoint will be double or triple or more.It will be a very
> big
> >> problem for me.How to understand it?
> >>
> >>
> >>
> >>
> >> 英文不太好，怕表达不清，再用中文叙述一下：
> >>
> >>
> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
> >>
> >>
> >> yours
> >> 大方
> >> 2021.09.11
> >
> >--
> >Sheng Wu 吴晟
> >
> >Apache SkyWalking
> >Apache Incubator
> >Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >Zipkin
> >Twitter, wusheng1108
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: A question about skywalking-kong
Posted by dafang <13...@163.com>.
Yeap,in our product design,some url is contains user id ,such as /hello/11230（/hello/{userId}）,and our user number is more than 10000000,in your example,it makes a lot trouble.How to deal this problem in any system in front of controller?
附：那所有在java controller之上的系统都会有这个问题吧，这个该怎么解决呢？
然后我还发现，对于span来说，我说的这些url是作为endpointName来存储和展示的，url中的动态参数如果不模板化替换，任何场景都会造成内存溢出问题
在 2021-09-11 10:51:36，"Sheng Wu" <wu...@gmail.com> 写道：
>This is not about Kong, it is for all services, even same for service mesh.
>This thing is very simple, URI is the entity ID to measure metrics. If you
>put parameter in the URI, such as /prod/123, then this is an entity. When
>you have 1million product IDs, then you face potential OOM easily.
>
>dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
>
>> Hi skywalking dev team:
>> I had read the source code in skywalking-kong,and I found that the
>> request_uri is wrote in request case.But I have heard that if the first
>> case if not set the url templated,the after will never set it again.If it's
>> set up,our endpoint will be double or triple or more.It will be a very big
>> problem for me.How to understand it?
>>
>>
>>
>>
>> 英文不太好，怕表达不清，再用中文叙述一下：
>>
>> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
>>
>>
>> yours
>> 大方
>> 2021.09.11
>
>--
>Sheng Wu 吴晟
>
>Apache SkyWalking
>Apache Incubator
>Apache ShardingSphere, ECharts, DolphinScheduler podlings
>Zipkin
>Twitter, wusheng1108
Re: A question about skywalking-kong
Posted by Sheng Wu <wu...@gmail.com>.
This is not about Kong, it is for all services, even same for service mesh.
This thing is very simple, URI is the entity ID to measure metrics. If you
put parameter in the URI, such as /prod/123, then this is an entity. When
you have 1million product IDs, then you face potential OOM easily.
dafang <13...@163.com>于2021年9月11日 周六上午9:46写道：
> Hi skywalking dev team:
> I had read the source code in skywalking-kong,and I found that the
> request_uri is wrote in request case.But I have heard that if the first
> case if not set the url templated,the after will never set it again.If it's
> set up,our endpoint will be double or triple or more.It will be a very big
> problem for me.How to understand it?
>
>
>
>
> 英文不太好，怕表达不清，再用中文叙述一下：
>
> 我最近阅读了skywalking-kong的源码，发现在记录request_url的时候，是直接记录的ngx.var.uri，这会导致url中携带动态参数，后面的系统会因为有了这个动态参数，而不进行模板化吧(记着之前听说过有这个机制)。这样就会导致系统存储的endpoint翻很多很多倍，如果打开了endpoint_alarm的话，那内存溢出估计会发生在skywalking启动的几分钟之内。这个问题开发团队有过专门的处理么？
>
>
> yours
> 大方
> 2021.09.11
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108