You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/12/28 01:36:41 UTC
[GitHub] [skywalking] buxingzhe opened a new issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
buxingzhe opened a new issue #6089:
URL: https://github.com/apache/skywalking/issues/6089
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [x] Question or discussion
Skywalking OAP server always gets stuck in frequent GC due to high memory usage in half an hour, and so does the UI.
From the live objects statistics output by JMap, there are a lot of endpoint relation metrics instances, just referring to my
comment. The scenario is we have many API endpoints whose url has path variables like user id, and we have millions of
different users. At the same time, I did the configuration to group the endpoints, but the issue was still there. So any proposal
the this issue?
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
___
### Bug
- Which version of SkyWalking, OS, and JRE?
8.3.0, Linux, Java 8
- Which company or project?
- What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.
___
### Requirement or improvement
- Please describe your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751565688
> If you have the parameterized mechanism, this should not be the concern of go2sky, right?
From go2sky code base , I see it just sets the raw API url path as the span operation name when the user doesn't set it explicitly.
So if we set the operation name with parameterized pattern explicitly, then the issue should be resolved?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751561860
One more I'd like to point out is our API gateway is developed by Golang, and we use go2sky SDK to do the trace instrumentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751546344
> Don't you have parameterized mechanism in your framework like SpringMVC? Such as /product/{id}, which could reduce the memory cost clearly.
Sure, we do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751562347
@wu-sheng what kind of scenario can lead to so many EndpointPoint*Metrics instances?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751546639
> > Don't you have parameterized mechanism in your framework like SpringMVC? Such as /product/{id}, which could reduce the memory cost clearly.
>
> Sure, we do.
Then why you still have countless issue? That is strange. The cache(persistent session) would only keep the data in one minute only for better performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6089:
URL: https://github.com/apache/skywalking/issues/6089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751546055
> The scenario is we have many API endpoints whose url has path variables like user id, and we have millions of
different users.
Don't you have parameterized mechanism in your framework like SpringMVC? Such as /product/{id}, which could reduce the memory cost clearly.
__
Which version of SkyWalking are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751555067
@zifeihan For gateway yes, but the backend provided the grouping mechanism to resolve this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751562373
> One more I'd like to point out is our API gateway is developed by Golang, and we use go2sky SDK to do the trace instrumentation.
If you have the parameterized mechanism, this should not be the concern of go2sky, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751566051
> So if we set the operation name with parameterized pattern explicitly, then the issue should be resolved?
From my understanding, it should be. If you don't have operation names of entry spans set randomly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751546365
> Which version of SkyWalking are you using?
8.3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] zifeihan commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
zifeihan commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751558204
> @zifeihan For gateway yes, but the backend provided the grouping mechanism to resolve this issue.
Ok, I get it now, thanks！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751548380
> > > Don't you have parameterized mechanism in your framework like SpringMVC? Such as /product/{id}, which could reduce the memory cost clearly.
> >
> >
> > Sure, we do.
>
> Then why you still have countless endpoints? That is strange. The cache(persistent session) would only keep the data in one minute only for better performance.
Actually, after I grouped the endpoints to a few groups, I can see the grouped endpoints as expected from the "Trace" tab.
but from "Dashboard" tab, in the "Current Endpoint" dropdown box we still see a lot of endpoint names including the concrete user id. By the way, how is the metrics instances count relation to the endpoint counts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] zifeihan commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
zifeihan commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751550934
@wu-sheng @buxingzhe
If there is gateway access, the interface of the gateway will not be serialized into the style of `/product/{id}`, because the forwarding of the gateway does not go to the step of serializing the endpoint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751548848
> By the way, how is the metrics instances count relation to the endpoint counts?
Nothing.
> I can see the grouped endpoints as expected from the "Trace" tab.
but from "Dashboard" tab, in the "Current Endpoint" dropdown box we still see a lot of endpoint names including the concrete user id.
They are sharing the same query. So, should not have any different. But they wouldn't be deleted before TTL reached.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751562429
> what kind of scenario can lead to so many EndpointPoint*Metrics instances?
Only the number of endpoint itself. Too many names.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751548619
Endpoint group configuration sample:
- service-name: api-gateway
rules:
- endpoint-name: /users/{userId}/properties
regex: .*\/users\/\w+\/properties
- endpoint-name: /weather/*
regex: .*\/weather\/.+
- endpoint-name: /users/{userId}/events
regex: .*\/users\/\w+\/events
- endpoint-name: /users/{userId}/events/*
regex: .*\/users\/\w+\/events\/.+
- endpoint-name: /*
regex: .*\/.+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751545292
Currently, I upgrade to use JDK 15, here is the GC logs:
[2020-12-28T01:40:05.953+0000][19 ][gc,phases ] GC(43522) Concurrent Mark 5257.144ms
[2020-12-28T01:40:05.954+0000][27 ][gc,phases ] GC(43522) Pause Mark End 0.540ms
[2020-12-28T01:40:05.954+0000][27 ][safepoint ] Safepoint "ZMarkEnd", Time since last: 5257202560 ns, Reaching safepoint: 100500 ns, A
t safepoint: 582341 ns, Total: 682841 ns
[2020-12-28T01:40:05.968+0000][19 ][gc,phases ] GC(43522) Concurrent Process Non-Strong References 14.212ms
[2020-12-28T01:40:05.968+0000][19 ][gc,phases ] GC(43522) Concurrent Reset Relocation Set 0.003ms
[2020-12-28T01:40:05.970+0000][19 ][gc,phases ] GC(43522) Concurrent Select Relocation Set 1.507ms
[2020-12-28T01:40:05.970+0000][27 ][gc,phases ] GC(43522) Pause Relocate Start 0.115ms
[2020-12-28T01:40:05.970+0000][27 ][safepoint ] Safepoint "ZRelocateStart", Time since last: 15788858 ns, Reaching safepoint: 93435 ns
, At safepoint: 142204 ns, Total: 235639 ns
[2020-12-28T01:40:05.971+0000][19 ][gc,phases ] GC(43522) Concurrent Relocate 1.060ms
[2020-12-28T01:40:05.971+0000][19 ][gc,load ] GC(43522) Load: 0.00/0.00/0.00
[2020-12-28T01:40:05.971+0000][19 ][gc,mmu ] GC(43522) MMU: 2ms/0.0%, 5ms/32.7%, 10ms/66.4%, 20ms/79.9%, 50ms/92.0%, 100ms/95.8%
[2020-12-28T01:40:05.971+0000][19 ][gc,marking ] GC(43522) Mark: 2 stripe(s), 2 proactive flush(es), 1 terminate flush(es), 1 completion(s), 0 continuation(s)
[2020-12-28T01:40:05.971+0000][19 ][gc,nmethod ] GC(43522) NMethods: 14445 registered, 2364 unregistered
[2020-12-28T01:40:05.971+0000][19 ][gc,metaspace] GC(43522) Metaspace: 67M used, 67M capacity, 68M committed, 1086M reserved
[2020-12-28T01:40:05.971+0000][19 ][gc,ref ] GC(43522) Soft: 1875 encountered, 223 discovered, 0 enqueued
[2020-12-28T01:40:05.971+0000][19 ][gc,ref ] GC(43522) Weak: 4746 encountered, 2743 discovered, 0 enqueued
[2020-12-28T01:40:05.971+0000][19 ][gc,ref ] GC(43522) Final: 813 encountered, 0 discovered, 0 enqueued
[2020-12-28T01:40:05.971+0000][19 ][gc,ref ] GC(43522) Phantom: 1018 encountered, 805 discovered, 0 enqueued
[2020-12-28T01:40:05.971+0000][19 ][gc,reloc ] GC(43522) Small Pages: 3953 / 7906M(97%), Empty: 2M(0%), Compacting: 12M(0%)->2M(0%)
[2020-12-28T01:40:05.971+0000][19 ][gc,reloc ] GC(43522) Medium Pages: 2 / 64M(1%), Empty: 0M(0%), Compacting: 0M(0%)->0M(0%)
[2020-12-28T01:40:05.971+0000][19 ][gc,reloc ] GC(43522) Large Pages: 20 / 170M(2%), Empty: 0M(0%), Compacting: 0M(0%)->0M(0%)
[2020-12-28T01:40:05.971+0000][19 ][gc,reloc ] GC(43522) Relocation: Successful
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Min Capacity: 8192M(100%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Max Capacity: 8192M(100%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Soft Max Capacity: 8192M(100%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Mark Start Mark End Relocate Start Relocate End High Low
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Capacity: 8192M (100%) 8192M (100%) 8192M (100%) 8192M (100%) 8192M (100%) 8192M (100%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Reserve: 44M (1%) 44M (1%) 44M (1%) 44M (1%) 44M (1%) 42M (1%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Free: 8M (0%) 2M (0%) 4M (0%) 8M (0%) 8M (0%) 0M (0%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Used: 8140M (99%) 8146M (99%) 8144M (99%) 8140M (99%) 8150M (99%) 8140M (99%)
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Live: - 6917M (84%) 6917M (84%) 6917M (84%) - -
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Allocated: - 6M (0%) 6M (0%) 14M (0%) - -
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Garbage: - 1222M (15%) 1220M (15%) 1216M (15%) - -
[2020-12-28T01:40:05.971+0000][19 ][gc,heap ] GC(43522) Reclaimed: - - 2M (0%) 6M (0%) - -
[2020-12-28T01:40:05.971+0000][19 ][gc ] GC(43522) Garbage Collection (Allocation Rate) 8140M(99%)->8140M(99%)
[2020-12-28T01:40:05.995+0000][19 ][gc,start ] GC(43523) Garbage Collection (High Usage)
[2020-12-28T01:40:05.995+0000][27 ][gc,phases ] GC(43523) Pause Mark Start 0.095ms
[2020-12-28T01:40:05.995+0000][27 ][safepoint ] Safepoint "ZMarkStart", Time since last: 25283212 ns, Reaching safepoint: 76998 ns, At safepoint: 116183 ns, Total: 193181 ns
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe commented on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe commented on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751544553
num #instances #bytes class name (module)
-------------------------------------------------------
1: 25053488 3446272760 [B (java.base@15.0.1)
2: 23396682 748693824 java.lang.String (java.base@15.0.1)
3: 1431571 454559584 [Ljava.util.HashMap$Node; (java.base@15.0.1)
4: 9210327 368413080 java.util.HashMap$Node (java.base@15.0.1)
5: 1616223 297385032 org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7InsertRequest
6: 2851893 182521152 java.util.HashMap (java.base@15.0.1)
7: 1170863 112402848 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointRelationPercentileMetrics
8: 1163043 111652128 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointRelationRespTimeMetrics
9: 1168276 102808288 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointRelationSlaMetrics
10: 1149526 101158288 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointRelationCpmMetrics
11: 2418159 96726360 java.util.LinkedList$Node (java.base@15.0.1)
12: 2849136 68379264 org.apache.skywalking.oap.server.core.analysis.metrics.DataTable
13: 1635244 65409760 org.elasticsearch.common.bytes.BytesArray
14: 1145388 64141728 org.apache.skywalking.oap.server.core.analysis.manual.relation.endpoint.EndpointRelationServerSideMetrics
15: 25116 52737264 [Ljava.lang.Object; (java.base@15.0.1)
16: 199757 14382504 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointAvgMetrics
17: 199695 14378040 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointPercentileMetrics
18: 241750 13538000 org.apache.skywalking.oap.server.core.alarm.provider.RunningRule$Window
19: 199695 12780480 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointSlaMetrics
20: 199197 12748608 org.apache.skywalking.oap.server.core.source.oal.rt.metrics.EndpointCpmMetrics
21: 278471 11138840 java.util.concurrent.ConcurrentHashMap$Node (java.base@15.0.1)
22: 243612 9744480 java.util.concurrent.locks.ReentrantLock$NonfairSync (java.base@15.0.1)
23: 242113 9684520 java.util.LinkedList (java.base@15.0.1)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng edited a comment on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
wu-sheng edited a comment on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751546639
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] buxingzhe edited a comment on issue #6089: OAP server gets stuck in frequent GC due to endpoint relation metrics high memory usage
Posted by GitBox <gi...@apache.org>.
buxingzhe edited a comment on issue #6089:
URL: https://github.com/apache/skywalking/issues/6089#issuecomment-751548619
Endpoint group configuration sample:
- service-name: api-gateway
rules:
- endpoint-name: /users/{userId}/properties
regex: .*\/users\/\w+\/properties
- endpoint-name: /weather/*
regex: .*\/weather\/.+
- endpoint-name: /users/{userId}/events
regex: .*\/users\/\w+\/events
- endpoint-name: /users/{userId}/events/*
regex: .*\/users\/\w+\/events\/.+
- endpoint-name: "/*"
regex: .*\/.+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org