You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by Sheng Wu <wu...@apache.org> on 2020/03/25 14:34:29 UTC
[Inventory Design Change] [WIP] Endpoint Inventory is going to be removed
Hi Dev team
If you are familiar with the latest OAP core, you will know, we register id
for the endpoint. But this design has the following issues
1. The register has low performance as it can't be run in parral even we
have clustering OAP. Even worse, in the cluster mode, the register is slower
2. The ID is designed as an integer, it could be run out in some time. We
don't have good idea to restore or extend it.
3. Endpoint including the parameter(such as, inside the URI) is always an
issue, and could only be improved in the agent plugin, but never could be
fixed.
The new changes are
1. Use `EndpointTraffic` to replace `EndpointInventory`. EndpointTraffic is
a manual metrics, not a register inventory.
2. ID register would not happen, and service_id + base64(endpoint name)
will be used as the entity id.
3. EndpointTraffic also keeps the time series feature like all other
metrics, so it could be removed by TTL.
4. EndpointRelation follows the changes of EndpointInevntory removal. It
keeps the source-service_id + base64(source endpoint name) and
dest-service_id + base64(dest endpoint name).
All the changes have been landed on the branch remove-endpoint-register
<https://github.com/apache/skywalking/tree/remove-endpoint-register> in
the main repo.
It passed the compiling locally, but not e2e and plugin change has not been
made(no endpoint register)
I am working on local debugging if you have interests, welcome to join me.
I expect this is the most important change since 5.0 core design and should
be landed in the 7.1.0
Sheng Wu 吴晟
Twitter, wusheng1108
OAP Core polish, especially storage level, was [Inventory Design
Change] [WIP] Endpoint Inventory is going to be removed
Posted by Sheng Wu <wu...@gmail.com>.
Hi Dev team
I submitted the PR[1] to do many changes to refactor codes, add tests and
polish definitions, especially for storage-related, including following
1. Add length definition with reasonable default value.
2. content of @Column has been renamed to storageOnly . I add this to
many fields as they are not being queried in any case. Should be better for
ES storage.
3. Merge H2 and MySQL columntype mapping back to consistent.
4. Remove @IDColumn.
5. Support @QueryUnifiedIndex.
6. Refactor the MySQL and H2 installers to use @Column and
@QueryUnifiedIndex definitions to create indices automatically. But the
index naming rule has been changed to entity*seq*IDX. seq is the
Increment Interger for every entity.
7. Support @MetricsExtension and insertOnly in the
MetricsPersistentWorker worker.
8. Optimize MetricsStreamProcessor
[1] https://github.com/apache/skywalking/pull/4587
Sheng Wu 吴晟
Twitter, wusheng1108
Sheng Wu <wu...@gmail.com> 于2020年3月28日周六 下午8:12写道：
> Hi
>
> The PR is ready.
>
> 1. All GitHub auto tests are passed
> 2. Documentations have been updated, including the protocol.
>
> I request to merge this PR, and in the next steps, I plan to
> 1. Optimize the @Column annotation, to add a) the index definition for
> H2/MySQL/InfluxDB. b) Add max length definition.
> 2. Support two metrics persistent type, ADD and ADD+UPDATE(current and
> default). ADD will improve the performance of Endpoint Traffic. There is no
> point to add it every time.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@gmail.com> 于2020年3月26日周四 上午10:30写道：
>
>> Glade to see the procedure is kicked off. Endpoints registration brings us
>> more pain than handy.
>> I would like to test the impaction of the performance and write
>> amplificant. Please keep us posted.
>>
>> Thanks, Hongtao.
>>
>> Sheng Wu <wu...@apache.org> 于2020年3月25日周三 下午10:34写道：
>>
>> > Hi Dev team
>> >
>> > If you are familiar with the latest OAP core, you will know, we
>> register id
>> > for the endpoint. But this design has the following issues
>> > 1. The register has low performance as it can't be run in parral even we
>> > have clustering OAP. Even worse, in the cluster mode, the register is
>> > slower
>> > 2. The ID is designed as an integer, it could be run out in some time.
>> We
>> > don't have good idea to restore or extend it.
>> > 3. Endpoint including the parameter(such as, inside the URI) is always
>> an
>> > issue, and could only be improved in the agent plugin, but never could
>> be
>> > fixed.
>> >
>> > The new changes are
>> > 1. Use `EndpointTraffic` to replace `EndpointInventory`.
>> EndpointTraffic is
>> > a manual metrics, not a register inventory.
>> > 2. ID register would not happen, and service_id + base64(endpoint name)
>> > will be used as the entity id.
>> > 3. EndpointTraffic also keeps the time series feature like all other
>> > metrics, so it could be removed by TTL.
>> > 4. EndpointRelation follows the changes of EndpointInevntory removal. It
>> > keeps the source-service_id + base64(source endpoint name) and
>> > dest-service_id + base64(dest endpoint name).
>> >
>> > All the changes have been landed on the branch remove-endpoint-register
>> > <https://github.com/apache/skywalking/tree/remove-endpoint-register>
>> in
>> > the main repo.
>> > It passed the compiling locally, but not e2e and plugin change has not
>> been
>> > made(no endpoint register)
>> > I am working on local debugging if you have interests, welcome to join
>> me.
>> > I expect this is the most important change since 5.0 core design and
>> should
>> > be landed in the 7.1.0
>> >
>> >
>> > Sheng Wu 吴晟
>> > Twitter, wusheng1108
>> >
>>
>>
>> --
>> Hongtao Gao
>>
>> Apache SkyWalking && Apache ShardingSphere
>> Twitter, @hanahmily
>>
>
Re: [Inventory Design Change] [WIP] Endpoint Inventory is going to be removed
Posted by Sheng Wu <wu...@gmail.com>.
Hi
The PR is ready.
1. All GitHub auto tests are passed
2. Documentations have been updated, including the protocol.
I request to merge this PR, and in the next steps, I plan to
1. Optimize the @Column annotation, to add a) the index definition for
H2/MySQL/InfluxDB. b) Add max length definition.
2. Support two metrics persistent type, ADD and ADD+UPDATE(current and
default). ADD will improve the performance of Endpoint Traffic. There is no
point to add it every time.
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@gmail.com> 于2020年3月26日周四 上午10:30写道：
> Glade to see the procedure is kicked off. Endpoints registration brings us
> more pain than handy.
> I would like to test the impaction of the performance and write
> amplificant. Please keep us posted.
>
> Thanks, Hongtao.
>
> Sheng Wu <wu...@apache.org> 于2020年3月25日周三 下午10:34写道：
>
> > Hi Dev team
> >
> > If you are familiar with the latest OAP core, you will know, we register
> id
> > for the endpoint. But this design has the following issues
> > 1. The register has low performance as it can't be run in parral even we
> > have clustering OAP. Even worse, in the cluster mode, the register is
> > slower
> > 2. The ID is designed as an integer, it could be run out in some time. We
> > don't have good idea to restore or extend it.
> > 3. Endpoint including the parameter(such as, inside the URI) is always an
> > issue, and could only be improved in the agent plugin, but never could be
> > fixed.
> >
> > The new changes are
> > 1. Use `EndpointTraffic` to replace `EndpointInventory`. EndpointTraffic
> is
> > a manual metrics, not a register inventory.
> > 2. ID register would not happen, and service_id + base64(endpoint name)
> > will be used as the entity id.
> > 3. EndpointTraffic also keeps the time series feature like all other
> > metrics, so it could be removed by TTL.
> > 4. EndpointRelation follows the changes of EndpointInevntory removal. It
> > keeps the source-service_id + base64(source endpoint name) and
> > dest-service_id + base64(dest endpoint name).
> >
> > All the changes have been landed on the branch remove-endpoint-register
> > <https://github.com/apache/skywalking/tree/remove-endpoint-register> in
> > the main repo.
> > It passed the compiling locally, but not e2e and plugin change has not
> been
> > made(no endpoint register)
> > I am working on local debugging if you have interests, welcome to join
> me.
> > I expect this is the most important change since 5.0 core design and
> should
> > be landed in the 7.1.0
> >
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
>
>
> --
> Hongtao Gao
>
> Apache SkyWalking && Apache ShardingSphere
> Twitter, @hanahmily
>
Re: [Inventory Design Change] [WIP] Endpoint Inventory is going to be removed
Posted by Hongtao Gao <ha...@gmail.com>.
Glade to see the procedure is kicked off. Endpoints registration brings us
more pain than handy.
I would like to test the impaction of the performance and write
amplificant. Please keep us posted.
Thanks, Hongtao.
Sheng Wu <wu...@apache.org> 于2020年3月25日周三 下午10:34写道：
> Hi Dev team
>
> If you are familiar with the latest OAP core, you will know, we register id
> for the endpoint. But this design has the following issues
> 1. The register has low performance as it can't be run in parral even we
> have clustering OAP. Even worse, in the cluster mode, the register is
> slower
> 2. The ID is designed as an integer, it could be run out in some time. We
> don't have good idea to restore or extend it.
> 3. Endpoint including the parameter(such as, inside the URI) is always an
> issue, and could only be improved in the agent plugin, but never could be
> fixed.
>
> The new changes are
> 1. Use `EndpointTraffic` to replace `EndpointInventory`. EndpointTraffic is
> a manual metrics, not a register inventory.
> 2. ID register would not happen, and service_id + base64(endpoint name)
> will be used as the entity id.
> 3. EndpointTraffic also keeps the time series feature like all other
> metrics, so it could be removed by TTL.
> 4. EndpointRelation follows the changes of EndpointInevntory removal. It
> keeps the source-service_id + base64(source endpoint name) and
> dest-service_id + base64(dest endpoint name).
>
> All the changes have been landed on the branch remove-endpoint-register
> <https://github.com/apache/skywalking/tree/remove-endpoint-register> in
> the main repo.
> It passed the compiling locally, but not e2e and plugin change has not been
> made(no endpoint register)
> I am working on local debugging if you have interests, welcome to join me.
> I expect this is the most important change since 5.0 core design and should
> be landed in the 7.1.0
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
--
Hongtao Gao
Apache SkyWalking && Apache ShardingSphere
Twitter, @hanahmily