You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by 成龙 <ja...@163.com> on 2019/12/11 10:24:02 UTC
How should we handle the cross-cloud invocation scenario?
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the compliance audit, and most of the business service is in Alibaba Public Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws NullPointerException becuase of referenceId exchange (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too much exception stack.
To solve these problems, we simply sync the register metadata across the cloud to make sure Skywalking in Finance Cloud can properly process the cross-cloud segment. In this case, we can collect all segment whether the trace is across cloud. Also we have the same traceId, we can find the same trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current probolems. So I really look forward to a better solution, or if someone has any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
|
Re: How should we handle the cross-cloud invocation scenario?
Posted by Sheng Wu <wu...@gmail.com>.
If I have the chance to Shenzhen recently, I will let you know.
Sheng Wu 吴晟
Twitter, wusheng1108
立龙(Lilong) <ja...@163.com> 于2019年12月12日周四 下午5:21写道：
> Shenzhen.
>
>
> It would be so great to meet you and discuss some tech problems f2f. In
> June this year, I met Mr. Gao hongtao at the GIAC conference in Shenzhen,
> which was also a lecture on Skywalking. But we were just starting to use
> version 6.1.0 in our test environment at this time. It's such a pity that
> we didn't have any questions.
>
>
> Looking forward to your visit to Shenzhen!
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
>
>
> On 12/12/2019 16:33，Sheng Wu<wu...@gmail.com> wrote：
> Sure. Look forward for that.
>
> Could you share where your company is located? The city I mean. I may could
> have a chance to stop there to discuss f2f next year if there is some tech
> meeting happens there.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 立龙(Lilong) <ja...@163.com> 于2019年12月12日周四 下午4:27写道：
>
> Yes, this solution bring in dependency of a new middleware, maybe a bit
> heavy, but it seems that to solve the cross-network problem, middleware is
> unavoidable.
>
>
> I have not yet implemeted the idea of using Redis in the morning into
> code. After I have finished one version of this solution and done enough
> tests, I will show the details of the whole design. I will try my best to
> optimize this implementation so that it can be adopted by the community.
>
>
> Thanks.
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
>
>
> On 12/12/2019 14:57，Sheng Wu<wu...@gmail.com> wrote：
> 成龙 <ja...@163.com> 于2019年12月12日周四 上午11:33写道：
>
>
>
> OK, I will try to write a configurable register implementation.
> Here are some of my thoughts:
> 1. Choose a central middleware to generate global ID and provide a
> channel to pub/sub the register message, Redis or Zookeeper should be
> fine, but consider the need for an external network address, Redis would be
> better.
> 2. All the register global ID is from Redis, and the register request
> will get to both clouds through pub/sub channel of Redis.
> 3. Of course, this is a configurable feature, we can open the config
> item when we truely faced with the cross-cloud problem.
>
>
> I wonder if those thoughts will be accepted by the communitry as a
> feature, we wanted to give all the changes back to the community, and
> follow the version iteration of the community.
>
>
> Thanks for considering this. I think the most challenge about this is, how
> to make this works in the community. A new storage provider? Maybe?
> In the community solution, we may need to keep the deployment solution as
> simple as possible, but I don't know whether this solution is, look like it
> includes Redis and some MQ tech.
> Could you provide more detailed proposals?
>
>
>
>
>
> By the way, if the trace is from different Skywalking but in different
> company, like we provide a payment service as a trading channel to another
> company, and we both use Skywalking. Is there a problem with this scenario?
>
>
> I think, technically, it is the same. Just make sure you are following your
> company's security requirements.
>
>
>
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
> On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
> OK. I think I got the point.
>
> In my mind, one new storage provider is a better idea.
>
> I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
> in the storage provider(ES's, I think).
> Then, you should try to write your own register implementation. Like
> 1. Use one place to generate ID, such as using a central Redis to generate
> ID. Across cloud accessible is the key point.
> 2. Replicated write to both clouds for metadata.
> 3. The query should keep your current change.
>
> You should be able to run all the things much more stable and easier than
> what you do today.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
>
>
>
> Thanks for your reply!
>
>
> Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
> and write a extra sync service. Details is as follows:
>
>
> 1. First of all, we add a config item to es storage, called "startIndex:
> ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
> this value(At the moment, we've written down a big enough number, it's not
> a good idea, we are optimizing,figure out how to make the id unique).
> Therefore, Skywalking in different cloud has a different sectional id.
> 2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
> data changes(only metadata), and send the data to Public Cloud
> RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
> Finance Cloud will receive the message, and execute the normal register
> process.
> 3. The message is carrying the ID which from cross-cloud, we use a map to
> storage the relationship of the normal register id and the cross-cloud id.
> 4. In the segment parse process, if the trace is across the cloud, we use
> the id-relationship-map to exchange the cross-cloud id to normal id, make
> Skywalking just like deal with the local trace.
> 5. Of course, also need to make a slight change to
> server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
> trace properly.
>
>
> I don't know if it's explained clearly. Thanks again for your reply!
>
>
>
> Lilong@iBOXCHAIN
> Middleware Developer
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
> Hi Lilong
>
> Thanks for sharing the details of your scenario.
> I may need a little more about the detail.
>
> I don't know how do you sync the metadata across the cloud? The register
> could happen concurrently, how could you guarantee the ID is unique in the
> global?
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
>
>
>
> Hello everyone,
>
>
> We know that trace, topology and metrics across cloud was not expect, but
> for a variety of reasons, we are now faced with the cross-cloud headache!
>
>
> Our core payment service is in Alibaba Finance Cloud because of the
> compliance audit, and most of the business service is in Alibaba Public
> Cloud because of the cost considerations.
> Here are some of our problems:
> 1. Each cloud has thier own Skywalking which responsible for the
> service tracing in their respective cloud regions;
> 2. The network between the cloud is separated, we can't send the trace
> segment to another directly;
> 3. Most of payment requests in Finance are came from Public Cloud, so
> we can't just ignore these traces;
> 4. Besides if the trace is across cloud, it will throws
> NullPointerException becuase of referenceId exchange
> (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
> much exception stack.
>
>
> To solve these problems, we simply sync the register metadata across the
> cloud to make sure Skywalking in Finance Cloud can properly process the
> cross-cloud segment. In this case, we can collect all segment whether the
> trace is across cloud. Also we have the same traceId, we can find the same
> trace in both side, even though it's just part of the whole trace.
>
>
> I know maybe this is not a good idea, but it can solve our current
> probolems. So I really look forward to a better solution, or if someone has
> any good ideas or suggestions on this, please let me know!
>
>
> Thanks.
>
>
> | |
> Lilong@iBOXCHAIN
> Middleware Developer
> |
> |
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> |
>
>
>
>
Re: How should we handle the cross-cloud invocation scenario?
Posted by "立龙 (Lilong)" <ja...@163.com>.
Shenzhen.
It would be so great to meet you and discuss some tech problems f2f. In June this year, I met Mr. Gao hongtao at the GIAC conference in Shenzhen, which was also a lecture on Skywalking. But we were just starting to use version 6.1.0 in our test environment at this time. It's such a pity that we didn't have any questions.
Looking forward to your visit to Shenzhen!
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/12/2019 16:33，Sheng Wu<wu...@gmail.com> wrote：
Sure. Look forward for that.
Could you share where your company is located? The city I mean. I may could
have a chance to stop there to discuss f2f next year if there is some tech
meeting happens there.
Sheng Wu 吴晟
Twitter, wusheng1108
立龙(Lilong) <ja...@163.com> 于2019年12月12日周四 下午4:27写道：
Yes, this solution bring in dependency of a new middleware, maybe a bit
heavy, but it seems that to solve the cross-network problem, middleware is
unavoidable.
I have not yet implemeted the idea of using Redis in the morning into
code. After I have finished one version of this solution and done enough
tests, I will show the details of the whole design. I will try my best to
optimize this implementation so that it can be adopted by the community.
Thanks.
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/12/2019 14:57，Sheng Wu<wu...@gmail.com> wrote：
成龙 <ja...@163.com> 于2019年12月12日周四 上午11:33写道：
OK, I will try to write a configurable register implementation.
Here are some of my thoughts:
1. Choose a central middleware to generate global ID and provide a
channel to pub/sub the register message, Redis or Zookeeper should be
fine, but consider the need for an external network address, Redis would be
better.
2. All the register global ID is from Redis, and the register request
will get to both clouds through pub/sub channel of Redis.
3. Of course, this is a configurable feature, we can open the config
item when we truely faced with the cross-cloud problem.
I wonder if those thoughts will be accepted by the communitry as a
feature, we wanted to give all the changes back to the community, and
follow the version iteration of the community.
Thanks for considering this. I think the most challenge about this is, how
to make this works in the community. A new storage provider? Maybe?
In the community solution, we may need to keep the deployment solution as
simple as possible, but I don't know whether this solution is, look like it
includes Redis and some MQ tech.
Could you provide more detailed proposals?
By the way, if the trace is from different Skywalking but in different
company, like we provide a payment service as a trading channel to another
company, and we both use Skywalking. Is there a problem with this scenario?
I think, technically, it is the same. Just make sure you are following your
company's security requirements.
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
OK. I think I got the point.
In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
in the storage provider(ES's, I think).
Then, you should try to write your own register implementation. Like
1. Use one place to generate ID, such as using a central Redis to generate
ID. Across cloud accessible is the key point.
2. Replicated write to both clouds for metadata.
3. The query should keep your current change.
You should be able to run all the things much more stable and easier than
what you do today.
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
Thanks for your reply!
Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
and write a extra sync service. Details is as follows:
1. First of all, we add a config item to es storage, called "startIndex:
${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
this value(At the moment, we've written down a big enough number, it's not
a good idea, we are optimizing,figure out how to make the id unique).
Therefore, Skywalking in different cloud has a different sectional id.
2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
data changes(only metadata), and send the data to Public Cloud
RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
Finance Cloud will receive the message, and execute the normal register
process.
3. The message is carrying the ID which from cross-cloud, we use a map to
storage the relationship of the normal register id and the cross-cloud id.
4. In the segment parse process, if the trace is across the cloud, we use
the id-relationship-map to exchange the cross-cloud id to normal id, make
Skywalking just like deal with the local trace.
5. Of course, also need to make a slight change to
server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
trace properly.
I don't know if it's explained clearly. Thanks again for your reply!
Lilong@iBOXCHAIN
Middleware Developer
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but
for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the
compliance audit, and most of the business service is in Alibaba Public
Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the
service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace
segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so
we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws
NullPointerException becuase of referenceId exchange
(ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
much exception stack.
To solve these problems, we simply sync the register metadata across the
cloud to make sure Skywalking in Finance Cloud can properly process the
cross-cloud segment. In this case, we can collect all segment whether the
trace is across cloud. Also we have the same traceId, we can find the same
trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current
probolems. So I really look forward to a better solution, or if someone has
any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
|
Re: How should we handle the cross-cloud invocation scenario?
Posted by Sheng Wu <wu...@gmail.com>.
Sure. Look forward for that.
Could you share where your company is located? The city I mean. I may could
have a chance to stop there to discuss f2f next year if there is some tech
meeting happens there.
Sheng Wu 吴晟
Twitter, wusheng1108
立龙(Lilong) <ja...@163.com> 于2019年12月12日周四 下午4:27写道：
> Yes, this solution bring in dependency of a new middleware, maybe a bit
> heavy, but it seems that to solve the cross-network problem, middleware is
> unavoidable.
>
>
> I have not yet implemeted the idea of using Redis in the morning into
> code. After I have finished one version of this solution and done enough
> tests, I will show the details of the whole design. I will try my best to
> optimize this implementation so that it can be adopted by the community.
>
>
> Thanks.
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
>
>
> On 12/12/2019 14:57，Sheng Wu<wu...@gmail.com> wrote：
> 成龙 <ja...@163.com> 于2019年12月12日周四 上午11:33写道：
>
>
>
> OK, I will try to write a configurable register implementation.
> Here are some of my thoughts:
> 1. Choose a central middleware to generate global ID and provide a
> channel to pub/sub the register message, Redis or Zookeeper should be
> fine, but consider the need for an external network address, Redis would be
> better.
> 2. All the register global ID is from Redis, and the register request
> will get to both clouds through pub/sub channel of Redis.
> 3. Of course, this is a configurable feature, we can open the config
> item when we truely faced with the cross-cloud problem.
>
>
> I wonder if those thoughts will be accepted by the communitry as a
> feature, we wanted to give all the changes back to the community, and
> follow the version iteration of the community.
>
>
> Thanks for considering this. I think the most challenge about this is, how
> to make this works in the community. A new storage provider? Maybe?
> In the community solution, we may need to keep the deployment solution as
> simple as possible, but I don't know whether this solution is, look like it
> includes Redis and some MQ tech.
> Could you provide more detailed proposals?
>
>
>
>
>
> By the way, if the trace is from different Skywalking but in different
> company, like we provide a payment service as a trading channel to another
> company, and we both use Skywalking. Is there a problem with this scenario?
>
>
> I think, technically, it is the same. Just make sure you are following your
> company's security requirements.
>
>
>
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
> On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
> OK. I think I got the point.
>
> In my mind, one new storage provider is a better idea.
>
> I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
> in the storage provider(ES's, I think).
> Then, you should try to write your own register implementation. Like
> 1. Use one place to generate ID, such as using a central Redis to generate
> ID. Across cloud accessible is the key point.
> 2. Replicated write to both clouds for metadata.
> 3. The query should keep your current change.
>
> You should be able to run all the things much more stable and easier than
> what you do today.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
>
>
>
> Thanks for your reply!
>
>
> Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
> and write a extra sync service. Details is as follows:
>
>
> 1. First of all, we add a config item to es storage, called "startIndex:
> ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
> this value(At the moment, we've written down a big enough number, it's not
> a good idea, we are optimizing,figure out how to make the id unique).
> Therefore, Skywalking in different cloud has a different sectional id.
> 2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
> data changes(only metadata), and send the data to Public Cloud
> RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
> Finance Cloud will receive the message, and execute the normal register
> process.
> 3. The message is carrying the ID which from cross-cloud, we use a map to
> storage the relationship of the normal register id and the cross-cloud id.
> 4. In the segment parse process, if the trace is across the cloud, we use
> the id-relationship-map to exchange the cross-cloud id to normal id, make
> Skywalking just like deal with the local trace.
> 5. Of course, also need to make a slight change to
> server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
> trace properly.
>
>
> I don't know if it's explained clearly. Thanks again for your reply!
>
>
>
> Lilong@iBOXCHAIN
> Middleware Developer
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
> Hi Lilong
>
> Thanks for sharing the details of your scenario.
> I may need a little more about the detail.
>
> I don't know how do you sync the metadata across the cloud? The register
> could happen concurrently, how could you guarantee the ID is unique in the
> global?
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
>
>
>
> Hello everyone,
>
>
> We know that trace, topology and metrics across cloud was not expect, but
> for a variety of reasons, we are now faced with the cross-cloud headache!
>
>
> Our core payment service is in Alibaba Finance Cloud because of the
> compliance audit, and most of the business service is in Alibaba Public
> Cloud because of the cost considerations.
> Here are some of our problems:
> 1. Each cloud has thier own Skywalking which responsible for the
> service tracing in their respective cloud regions;
> 2. The network between the cloud is separated, we can't send the trace
> segment to another directly;
> 3. Most of payment requests in Finance are came from Public Cloud, so
> we can't just ignore these traces;
> 4. Besides if the trace is across cloud, it will throws
> NullPointerException becuase of referenceId exchange
> (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
> much exception stack.
>
>
> To solve these problems, we simply sync the register metadata across the
> cloud to make sure Skywalking in Finance Cloud can properly process the
> cross-cloud segment. In this case, we can collect all segment whether the
> trace is across cloud. Also we have the same traceId, we can find the same
> trace in both side, even though it's just part of the whole trace.
>
>
> I know maybe this is not a good idea, but it can solve our current
> probolems. So I really look forward to a better solution, or if someone has
> any good ideas or suggestions on this, please let me know!
>
>
> Thanks.
>
>
> | |
> Lilong@iBOXCHAIN
> Middleware Developer
> |
> |
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> |
>
>
>
Re: How should we handle the cross-cloud invocation scenario?
Posted by "立龙 (Lilong)" <ja...@163.com>.
Yes, this solution bring in dependency of a new middleware, maybe a bit heavy, but it seems that to solve the cross-network problem, middleware is unavoidable.
I have not yet implemeted the idea of using Redis in the morning into code. After I have finished one version of this solution and done enough tests, I will show the details of the whole design. I will try my best to optimize this implementation so that it can be adopted by the community.
Thanks.
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/12/2019 14:57，Sheng Wu<wu...@gmail.com> wrote：
成龙 <ja...@163.com> 于2019年12月12日周四 上午11:33写道：
OK, I will try to write a configurable register implementation.
Here are some of my thoughts:
1. Choose a central middleware to generate global ID and provide a
channel to pub/sub the register message, Redis or Zookeeper should be
fine, but consider the need for an external network address, Redis would be
better.
2. All the register global ID is from Redis, and the register request
will get to both clouds through pub/sub channel of Redis.
3. Of course, this is a configurable feature, we can open the config
item when we truely faced with the cross-cloud problem.
I wonder if those thoughts will be accepted by the communitry as a
feature, we wanted to give all the changes back to the community, and
follow the version iteration of the community.
Thanks for considering this. I think the most challenge about this is, how
to make this works in the community. A new storage provider? Maybe?
In the community solution, we may need to keep the deployment solution as
simple as possible, but I don't know whether this solution is, look like it
includes Redis and some MQ tech.
Could you provide more detailed proposals?
By the way, if the trace is from different Skywalking but in different
company, like we provide a payment service as a trading channel to another
company, and we both use Skywalking. Is there a problem with this scenario?
I think, technically, it is the same. Just make sure you are following your
company's security requirements.
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
OK. I think I got the point.
In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
in the storage provider(ES's, I think).
Then, you should try to write your own register implementation. Like
1. Use one place to generate ID, such as using a central Redis to generate
ID. Across cloud accessible is the key point.
2. Replicated write to both clouds for metadata.
3. The query should keep your current change.
You should be able to run all the things much more stable and easier than
what you do today.
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
Thanks for your reply!
Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
and write a extra sync service. Details is as follows:
1. First of all, we add a config item to es storage, called "startIndex:
${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
this value(At the moment, we've written down a big enough number, it's not
a good idea, we are optimizing,figure out how to make the id unique).
Therefore, Skywalking in different cloud has a different sectional id.
2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
data changes(only metadata), and send the data to Public Cloud
RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
Finance Cloud will receive the message, and execute the normal register
process.
3. The message is carrying the ID which from cross-cloud, we use a map to
storage the relationship of the normal register id and the cross-cloud id.
4. In the segment parse process, if the trace is across the cloud, we use
the id-relationship-map to exchange the cross-cloud id to normal id, make
Skywalking just like deal with the local trace.
5. Of course, also need to make a slight change to
server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
trace properly.
I don't know if it's explained clearly. Thanks again for your reply!
Lilong@iBOXCHAIN
Middleware Developer
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but
for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the
compliance audit, and most of the business service is in Alibaba Public
Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the
service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace
segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so
we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws
NullPointerException becuase of referenceId exchange
(ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
much exception stack.
To solve these problems, we simply sync the register metadata across the
cloud to make sure Skywalking in Finance Cloud can properly process the
cross-cloud segment. In this case, we can collect all segment whether the
trace is across cloud. Also we have the same traceId, we can find the same
trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current
probolems. So I really look forward to a better solution, or if someone has
any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
|
Re: How should we handle the cross-cloud invocation scenario?
Posted by Sheng Wu <wu...@gmail.com>.
成龙 <ja...@163.com> 于2019年12月12日周四 上午11:33写道：
>
>
> OK, I will try to write a configurable register implementation.
> Here are some of my thoughts:
> 1. Choose a central middleware to generate global ID and provide a
> channel to pub/sub the register message, Redis or Zookeeper should be
> fine, but consider the need for an external network address, Redis would be
> better.
> 2. All the register global ID is from Redis, and the register request
> will get to both clouds through pub/sub channel of Redis.
> 3. Of course, this is a configurable feature, we can open the config
> item when we truely faced with the cross-cloud problem.
>
>
> I wonder if those thoughts will be accepted by the communitry as a
> feature, we wanted to give all the changes back to the community, and
> follow the version iteration of the community.
>
Thanks for considering this. I think the most challenge about this is, how
to make this works in the community. A new storage provider? Maybe?
In the community solution, we may need to keep the deployment solution as
simple as possible, but I don't know whether this solution is, look like it
includes Redis and some MQ tech.
Could you provide more detailed proposals?
>
>
> By the way, if the trace is from different Skywalking but in different
> company, like we provide a payment service as a trading channel to another
> company, and we both use Skywalking. Is there a problem with this scenario?
>
I think, technically, it is the same. Just make sure you are following your
company's security requirements.
> _______________________________
> Lilong@iBOXCHAIN
> Middleware Developer
> On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
> OK. I think I got the point.
>
> In my mind, one new storage provider is a better idea.
>
> I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
> in the storage provider(ES's, I think).
> Then, you should try to write your own register implementation. Like
> 1. Use one place to generate ID, such as using a central Redis to generate
> ID. Across cloud accessible is the key point.
> 2. Replicated write to both clouds for metadata.
> 3. The query should keep your current change.
>
> You should be able to run all the things much more stable and easier than
> what you do today.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
>
>
>
> Thanks for your reply!
>
>
> Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
> and write a extra sync service. Details is as follows:
>
>
> 1. First of all, we add a config item to es storage, called "startIndex:
> ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
> this value(At the moment, we've written down a big enough number, it's not
> a good idea, we are optimizing,figure out how to make the id unique).
> Therefore, Skywalking in different cloud has a different sectional id.
> 2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
> data changes(only metadata), and send the data to Public Cloud
> RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
> Finance Cloud will receive the message, and execute the normal register
> process.
> 3. The message is carrying the ID which from cross-cloud, we use a map to
> storage the relationship of the normal register id and the cross-cloud id.
> 4. In the segment parse process, if the trace is across the cloud, we use
> the id-relationship-map to exchange the cross-cloud id to normal id, make
> Skywalking just like deal with the local trace.
> 5. Of course, also need to make a slight change to
> server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
> trace properly.
>
>
> I don't know if it's explained clearly. Thanks again for your reply!
>
>
>
> Lilong@iBOXCHAIN
> Middleware Developer
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
> Hi Lilong
>
> Thanks for sharing the details of your scenario.
> I may need a little more about the detail.
>
> I don't know how do you sync the metadata across the cloud? The register
> could happen concurrently, how could you guarantee the ID is unique in the
> global?
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
>
>
>
> Hello everyone,
>
>
> We know that trace, topology and metrics across cloud was not expect, but
> for a variety of reasons, we are now faced with the cross-cloud headache!
>
>
> Our core payment service is in Alibaba Finance Cloud because of the
> compliance audit, and most of the business service is in Alibaba Public
> Cloud because of the cost considerations.
> Here are some of our problems:
> 1. Each cloud has thier own Skywalking which responsible for the
> service tracing in their respective cloud regions;
> 2. The network between the cloud is separated, we can't send the trace
> segment to another directly;
> 3. Most of payment requests in Finance are came from Public Cloud, so
> we can't just ignore these traces;
> 4. Besides if the trace is across cloud, it will throws
> NullPointerException becuase of referenceId exchange
> (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
> much exception stack.
>
>
> To solve these problems, we simply sync the register metadata across the
> cloud to make sure Skywalking in Finance Cloud can properly process the
> cross-cloud segment. In this case, we can collect all segment whether the
> trace is across cloud. Also we have the same traceId, we can find the same
> trace in both side, even though it's just part of the whole trace.
>
>
> I know maybe this is not a good idea, but it can solve our current
> probolems. So I really look forward to a better solution, or if someone has
> any good ideas or suggestions on this, please let me know!
>
>
> Thanks.
>
>
> | |
> Lilong@iBOXCHAIN
> Middleware Developer
> |
> |
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> |
>
>
Re: How should we handle the cross-cloud invocation scenario?
Posted by 成龙 <ja...@163.com>.
OK, I will try to write a configurable register implementation.
Here are some of my thoughts:
1. Choose a central middleware to generate global ID and provide a channel to pub/sub the register message, Redis or Zookeeper should be fine, but consider the need for an external network address, Redis would be better.
2. All the register global ID is from Redis, and the register request will get to both clouds through pub/sub channel of Redis.
3. Of course, this is a configurable feature, we can open the config item when we truely faced with the cross-cloud problem.
I wonder if those thoughts will be accepted by the communitry as a feature, we wanted to give all the changes back to the community, and follow the version iteration of the community.
By the way, if the trace is from different Skywalking but in different company, like we provide a payment service as a trading channel to another company, and we both use Skywalking. Is there a problem with this scenario?
_______________________________
Lilong@iBOXCHAIN
Middleware Developer
On 12/11/2019 21:48，Sheng Wu<wu...@gmail.com> wrote：
OK. I think I got the point.
In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
in the storage provider(ES's, I think).
Then, you should try to write your own register implementation. Like
1. Use one place to generate ID, such as using a central Redis to generate
ID. Across cloud accessible is the key point.
2. Replicated write to both clouds for metadata.
3. The query should keep your current change.
You should be able to run all the things much more stable and easier than
what you do today.
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
Thanks for your reply!
Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
and write a extra sync service. Details is as follows:
1. First of all, we add a config item to es storage, called "startIndex:
${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
this value(At the moment, we've written down a big enough number, it's not
a good idea, we are optimizing,figure out how to make the id unique).
Therefore, Skywalking in different cloud has a different sectional id.
2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
data changes(only metadata), and send the data to Public Cloud
RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
Finance Cloud will receive the message, and execute the normal register
process.
3. The message is carrying the ID which from cross-cloud, we use a map to
storage the relationship of the normal register id and the cross-cloud id.
4. In the segment parse process, if the trace is across the cloud, we use
the id-relationship-map to exchange the cross-cloud id to normal id, make
Skywalking just like deal with the local trace.
5. Of course, also need to make a slight change to
server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
trace properly.
I don't know if it's explained clearly. Thanks again for your reply!
Lilong@iBOXCHAIN
Middleware Developer
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but
for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the
compliance audit, and most of the business service is in Alibaba Public
Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the
service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace
segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so
we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws
NullPointerException becuase of referenceId exchange
(ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
much exception stack.
To solve these problems, we simply sync the register metadata across the
cloud to make sure Skywalking in Finance Cloud can properly process the
cross-cloud segment. In this case, we can collect all segment whether the
trace is across cloud. Also we have the same traceId, we can find the same
trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current
probolems. So I really look forward to a better solution, or if someone has
any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
|
Re: How should we handle the cross-cloud invocation scenario?
Posted by Sheng Wu <wu...@gmail.com>.
OK. I think I got the point.
In my mind, one new storage provider is a better idea.
I recommend you to read IRegisterLockDAO and IRegisterDAO implementations
in the storage provider(ES's, I think).
Then, you should try to write your own register implementation. Like
1. Use one place to generate ID, such as using a central Redis to generate
ID. Across cloud accessible is the key point.
2. Replicated write to both clouds for metadata.
3. The query should keep your current change.
You should be able to run all the things much more stable and easier than
what you do today.
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午9:40写道：
>
>
> Thanks for your reply!
>
>
> Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server,
> and write a extra sync service. Details is as follows:
>
>
> 1. First of all, we add a config item to es storage, called "startIndex:
> ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from
> this value(At the moment, we've written down a big enough number, it's not
> a good idea, we are optimizing,figure out how to make the id unique).
> Therefore, Skywalking in different cloud has a different sectional id.
> 2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es
> data changes(only metadata), and send the data to Public Cloud
> RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in
> Finance Cloud will receive the message, and execute the normal register
> process.
> 3. The message is carrying the ID which from cross-cloud, we use a map to
> storage the relationship of the normal register id and the cross-cloud id.
> 4. In the segment parse process, if the trace is across the cloud, we use
> the id-relationship-map to exchange the cross-cloud id to normal id, make
> Skywalking just like deal with the local trace.
> 5. Of course, also need to make a slight change to
> server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud
> trace properly.
>
>
> I don't know if it's explained clearly. Thanks again for your reply!
>
>
>
> Lilong@iBOXCHAIN
> Middleware Developer
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
> Hi Lilong
>
> Thanks for sharing the details of your scenario.
> I may need a little more about the detail.
>
> I don't know how do you sync the metadata across the cloud? The register
> could happen concurrently, how could you guarantee the ID is unique in the
> global?
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> 成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
>
>
>
> Hello everyone,
>
>
> We know that trace, topology and metrics across cloud was not expect, but
> for a variety of reasons, we are now faced with the cross-cloud headache!
>
>
> Our core payment service is in Alibaba Finance Cloud because of the
> compliance audit, and most of the business service is in Alibaba Public
> Cloud because of the cost considerations.
> Here are some of our problems:
> 1. Each cloud has thier own Skywalking which responsible for the
> service tracing in their respective cloud regions;
> 2. The network between the cloud is separated, we can't send the trace
> segment to another directly;
> 3. Most of payment requests in Finance are came from Public Cloud, so
> we can't just ignore these traces;
> 4. Besides if the trace is across cloud, it will throws
> NullPointerException becuase of referenceId exchange
> (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
> much exception stack.
>
>
> To solve these problems, we simply sync the register metadata across the
> cloud to make sure Skywalking in Finance Cloud can properly process the
> cross-cloud segment. In this case, we can collect all segment whether the
> trace is across cloud. Also we have the same traceId, we can find the same
> trace in both side, even though it's just part of the whole trace.
>
>
> I know maybe this is not a good idea, but it can solve our current
> probolems. So I really look forward to a better solution, or if someone has
> any good ideas or suggestions on this, please let me know!
>
>
> Thanks.
>
>
> | |
> Lilong@iBOXCHAIN
> Middleware Developer
> |
> |
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> |
>
Re: How should we handle the cross-cloud invocation scenario?
Posted by 成龙 <ja...@163.com>.
Thanks for your reply!
Oh, yes, to avoid ID conflicts, we made some slight changes to OAP server, and write a extra sync service. Details is as follows:
1. First of all, we add a config item to es storage, called "startIndex: ${SW_STORAGE_ES_START_INDEX:100000}", means all register ID is start from this value(At the moment, we've written down a big enough number, it's not a good idea, we are optimizing,figure out how to make the id unique). Therefore, Skywalking in different cloud has a different sectional id.
2. The extra sync service in Public Cloudwill subscribe thePublic Cloud es data changes(only metadata), and send the data to Public Cloud RocketMQ(Internet HTTP Endpoint), meanwhile the extra sync service in Finance Cloud will receive the message, and execute the normal register process.
3. The message is carrying the ID which from cross-cloud, we use a map to storage the relationship of the normal register id and the cross-cloud id.
4. In the segment parse process, if the trace is across the cloud, we use the id-relationship-map to exchange the cross-cloud id to normal id, make Skywalking just like deal with the local trace.
5. Of course, also need to make a slight change to server-query-plugin#TraceQuery, make sure the UI shows the cross-cloud trace properly.
I don't know if it's explained clearly. Thanks again for your reply!
Lilong@iBOXCHAIN
Middleware Developer
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
On 12/11/2019 20:31，Sheng Wu<wu...@gmail.com> wrote：
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
Hello everyone,
We know that trace, topology and metrics across cloud was not expect, but
for a variety of reasons, we are now faced with the cross-cloud headache!
Our core payment service is in Alibaba Finance Cloud because of the
compliance audit, and most of the business service is in Alibaba Public
Cloud because of the cost considerations.
Here are some of our problems:
1. Each cloud has thier own Skywalking which responsible for the
service tracing in their respective cloud regions;
2. The network between the cloud is separated, we can't send the trace
segment to another directly;
3. Most of payment requests in Finance are came from Public Cloud, so
we can't just ignore these traces;
4. Besides if the trace is across cloud, it will throws
NullPointerException becuase of referenceId exchange
(ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
much exception stack.
To solve these problems, we simply sync the register metadata across the
cloud to make sure Skywalking in Finance Cloud can properly process the
cross-cloud segment. In this case, we can collect all segment whether the
trace is across cloud. Also we have the same traceId, we can find the same
trace in both side, even though it's just part of the whole trace.
I know maybe this is not a good idea, but it can solve our current
probolems. So I really look forward to a better solution, or if someone has
any good ideas or suggestions on this, please let me know!
Thanks.
| |
Lilong@iBOXCHAIN
Middleware Developer
|
|
Shenzhen iBOXCHAIN technology co. LTD
jackiechan1981@163.com
|
Re: How should we handle the cross-cloud invocation scenario?
Posted by Sheng Wu <wu...@gmail.com>.
Hi Lilong
Thanks for sharing the details of your scenario.
I may need a little more about the detail.
I don't know how do you sync the metadata across the cloud? The register
could happen concurrently, how could you guarantee the ID is unique in the
global?
Sheng Wu 吴晟
Twitter, wusheng1108
成龙 <ja...@163.com> 于2019年12月11日周三 下午6:24写道：
>
>
> Hello everyone,
>
>
> We know that trace, topology and metrics across cloud was not expect, but
> for a variety of reasons, we are now faced with the cross-cloud headache!
>
>
> Our core payment service is in Alibaba Finance Cloud because of the
> compliance audit, and most of the business service is in Alibaba Public
> Cloud because of the cost considerations.
> Here are some of our problems:
> 1. Each cloud has thier own Skywalking which responsible for the
> service tracing in their respective cloud regions;
> 2. The network between the cloud is separated, we can't send the trace
> segment to another directly;
> 3. Most of payment requests in Finance are came from Public Cloud, so
> we can't just ignore these traces;
> 4. Besides if the trace is across cloud, it will throws
> NullPointerException becuase of referenceId exchange
> (ReferenceIdExchanger.java:60, baseed on 6.5.0), OAP server will print too
> much exception stack.
>
>
> To solve these problems, we simply sync the register metadata across the
> cloud to make sure Skywalking in Finance Cloud can properly process the
> cross-cloud segment. In this case, we can collect all segment whether the
> trace is across cloud. Also we have the same traceId, we can find the same
> trace in both side, even though it's just part of the whole trace.
>
>
> I know maybe this is not a good idea, but it can solve our current
> probolems. So I really look forward to a better solution, or if someone has
> any good ideas or suggestions on this, please let me know!
>
>
> Thanks.
>
>
> | |
> Lilong@iBOXCHAIN
> Middleware Developer
> |
> |
> Shenzhen iBOXCHAIN technology co. LTD
> jackiechan1981@163.com
> |