You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by han liu <mr...@gmail.com> on 2020/03/04 09:06:07 UTC
A new proposal of message transmission
Hi dev team,
I want to provide a message transmission future, which can be used for
message transmission in projects that already use skywalking.
This future of data transmission between multiple instances in the same
trace can be achieved by adding a new API of information transmission in
each language level, which does not need a dependency on the existing
interface protocol.
For example, the execution process of a trace is A -> B -> C, Set transfer
content in A service, B and C services can obtain data through API.
This future is to use the message passing future of Skywaling, addition an
extension function. I'm not quite sure if it's appropriate to add it to
Skywalking?
Currently, it can be applied to these scenarios:
1. For the existing old system, It's not convenient to revise the data of
the existing protocol. Through embedding transparent data, downstream
services can get data information.
2. In the entry layer, such as zuul or other gateway services,
transparently transmits user identity or other customized information to
the downstream services.
Implementation plan:
1. Store transmission data as Map in the current segment.
2. Create a new protocol header for data transmission (JSON)
3. When the downstream system receives the new protocol, it will transform
the data back to Map and store to the segment
4. I think this implementation should apply to trace and ignored segment,
otherwise, transmission data maybe will be lost.
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Let's hold the implementation plan for a while.
I want to hear what other PMC/committer think about this?
Is this useful or making sense to you?
I have heard this requirement several times, but no one had proposed a
complete plan like this and was trying to implement and contribute this.
So, before all these happen, please provide feedback.
Sheng Wu 吴晟
Twitter, wusheng1108
han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> Hi dev team,
>
> I want to provide a message transmission future, which can be used for
> message transmission in projects that already use skywalking.
> This future of data transmission between multiple instances in the same
> trace can be achieved by adding a new API of information transmission in
> each language level, which does not need a dependency on the existing
> interface protocol.
> For example, the execution process of a trace is A -> B -> C, Set transfer
> content in A service, B and C services can obtain data through API.
> This future is to use the message passing future of Skywaling, addition an
> extension function. I'm not quite sure if it's appropriate to add it to
> Skywalking?
>
> Currently, it can be applied to these scenarios:
> 1. For the existing old system, It's not convenient to revise the data of
> the existing protocol. Through embedding transparent data, downstream
> services can get data information.
> 2. In the entry layer, such as zuul or other gateway services,
> transparently transmits user identity or other customized information to
> the downstream services.
>
> Implementation plan:
> 1. Store transmission data as Map in the current segment.
> 2. Create a new protocol header for data transmission (JSON)
> 3. When the downstream system receives the new protocol, it will transform
> the data back to Map and store to the segment
> 4. I think this implementation should apply to trace and ignored segment,
> otherwise, transmission data maybe will be lost.
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
han liu <mr...@gmail.com> 于2020年3月5日周四 下午8:03写道：
> Hi Sheng Wu
>
> I totally agree with it, and I think we only need to provide some API is
> enough . Log level does not need to provide support.
>
Agree. Once this begins, I would say please provide a pull request to add
the header protocol description to the repo. After that, the other would be
easier.
We could encourage other agents to support this too.
Sheng Wu 吴晟
Twitter, wusheng1108
>
> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午4:58写道：
>
> > Hi Han Liu
> >
> > Positive feedback from Zipkin team, Read this from Adrian, Zipkin lead.
> > https://twitter.com/adrianfcole/status/1235424536608174080
> > Some similar features in the Zipkin Java SDK, brave project.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> >
> > > Hi Gao
> > >
> > > I am still confusing. If b3 used, then trace is the trace, there is
> > > nothing related to what we do.
> > > If application is instrumented by Zipkin SDK, then it is Zipkin's
> > > transmission capability, not us.
> > >
> > > In the proposal, it is clear that our agent is going to propagate
> things
> > > other injected and propagate them without touch. I can't see the link
> > > between this and your mesh case.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> > >
> > >> Let me clarify what I'm talking about.
> > >>
> > >> In the Istio demo, just like Bookinfo, all of the apps implement
> Zipkin
> > >> propagation, which means they can extract trace context from
> downstream
> > >> and
> > >> inject it to upstream.
> > >>
> > >> I want downside envoy sidecar instrumented by SkyWalking can wrap sw
> > trace
> > >> context into B3 context to bypass target app/service to upside
> sidecar.
> > >>
> > >> When it happens, skywalking can trace application instrumented only by
> > >> Zipkin in service mesh scenario.
> > >>
> > >> I prefer to Zipkin that because it seems the actual standard tracing
> > >> protocol in the SM world. And we leverage it as a lower-level protocol
> > to
> > >> figure out tracing context overlay.
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
> > wrote:
> > >>
> > >> > Hi Gao
> > >> >
> > >> > Look through your reply, are you talking about propagating zipkin
> > >> context
> > >> > or skywalking context? Your case is not clear to me.
> > >> >
> > >> > Sheng Wu 吴晟
> > >> > Twitter, wusheng1108
> > >> >
> > >> >
> > >> > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > >> >
> > >> > > Look good to me. I think this feature is also fantastic to the
> > Service
> > >> > Mesh
> > >> > > scenario.
> > >> > > For now, OAP ingests send and receive messages from the sidecar
> and
> > >> uses
> > >> > > statistic analysis to link them to a topology. There's no trace
> > here.
> > >> > >
> > >> > > If we implement the transmission protocol, the service workload
> can
> > >> > bypass
> > >> > > skywalking protocol, sidecar creates a segment and sinks it to OAP
> > >> > instead.
> > >> > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace
> is
> > >> > > connected by the application who implement Zipkin protocol. I know
> > it
> > >> may
> > >> > > be
> > >> > > a big challenge for us, but from my perspective, it's not
> impossible
> > >> > > totally.
> > >> > >
> > >> > > Thanks, Hongtao
> > >> > >
> > >> > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > >> > >
> > >> > > > Hi dev team,
> > >> > > >
> > >> > > > I want to provide a message transmission future, which can be
> used
> > >> for
> > >> > > > message transmission in projects that already use skywalking.
> > >> > > > This future of data transmission between multiple instances in
> the
> > >> same
> > >> > > > trace can be achieved by adding a new API of information
> > >> transmission
> > >> > in
> > >> > > > each language level, which does not need a dependency on the
> > >> existing
> > >> > > > interface protocol.
> > >> > > > For example, the execution process of a trace is A -> B -> C,
> Set
> > >> > > transfer
> > >> > > > content in A service, B and C services can obtain data through
> > API.
> > >> > > > This future is to use the message passing future of Skywaling,
> > >> addition
> > >> > > an
> > >> > > > extension function. I'm not quite sure if it's appropriate to
> add
> > >> it to
> > >> > > > Skywalking?
> > >> > > >
> > >> > > > Currently, it can be applied to these scenarios:
> > >> > > > 1. For the existing old system, It's not convenient to revise
> the
> > >> data
> > >> > of
> > >> > > > the existing protocol. Through embedding transparent data,
> > >> downstream
> > >> > > > services can get data information.
> > >> > > > 2. In the entry layer, such as zuul or other gateway services,
> > >> > > > transparently transmits user identity or other customized
> > >> information
> > >> > to
> > >> > > > the downstream services.
> > >> > > >
> > >> > > > Implementation plan:
> > >> > > > 1. Store transmission data as Map in the current segment.
> > >> > > > 2. Create a new protocol header for data transmission (JSON)
> > >> > > > 3. When the downstream system receives the new protocol, it will
> > >> > > transform
> > >> > > > the data back to Map and store to the segment
> > >> > > > 4. I think this implementation should apply to trace and ignored
> > >> > segment,
> > >> > > > otherwise, transmission data maybe will be lost.
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Hongtao Gao
> > >> > >
> > >> > > Apache SkyWalking && Apache ShardingSphere
> > >> > > Twitter, @hanahmily
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: A new proposal of message transmission
Posted by han liu <mr...@gmail.com>.
Hi Sheng Wu
I totally agree with it, and I think we only need to provide some API is
enough . Log level does not need to provide support.
Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午4:58写道：
> Hi Han Liu
>
> Positive feedback from Zipkin team, Read this from Adrian, Zipkin lead.
> https://twitter.com/adrianfcole/status/1235424536608174080
> Some similar features in the Zipkin Java SDK, brave project.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
>
> > Hi Gao
> >
> > I am still confusing. If b3 used, then trace is the trace, there is
> > nothing related to what we do.
> > If application is instrumented by Zipkin SDK, then it is Zipkin's
> > transmission capability, not us.
> >
> > In the proposal, it is clear that our agent is going to propagate things
> > other injected and propagate them without touch. I can't see the link
> > between this and your mesh case.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> >
> >> Let me clarify what I'm talking about.
> >>
> >> In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
> >> propagation, which means they can extract trace context from downstream
> >> and
> >> inject it to upstream.
> >>
> >> I want downside envoy sidecar instrumented by SkyWalking can wrap sw
> trace
> >> context into B3 context to bypass target app/service to upside sidecar.
> >>
> >> When it happens, skywalking can trace application instrumented only by
> >> Zipkin in service mesh scenario.
> >>
> >> I prefer to Zipkin that because it seems the actual standard tracing
> >> protocol in the SM world. And we leverage it as a lower-level protocol
> to
> >> figure out tracing context overlay.
> >>
> >>
> >>
> >>
> >> On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
> wrote:
> >>
> >> > Hi Gao
> >> >
> >> > Look through your reply, are you talking about propagating zipkin
> >> context
> >> > or skywalking context? Your case is not clear to me.
> >> >
> >> > Sheng Wu 吴晟
> >> > Twitter, wusheng1108
> >> >
> >> >
> >> > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> >> >
> >> > > Look good to me. I think this feature is also fantastic to the
> Service
> >> > Mesh
> >> > > scenario.
> >> > > For now, OAP ingests send and receive messages from the sidecar and
> >> uses
> >> > > statistic analysis to link them to a topology. There's no trace
> here.
> >> > >
> >> > > If we implement the transmission protocol, the service workload can
> >> > bypass
> >> > > skywalking protocol, sidecar creates a segment and sinks it to OAP
> >> > instead.
> >> > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> >> > > connected by the application who implement Zipkin protocol. I know
> it
> >> may
> >> > > be
> >> > > a big challenge for us, but from my perspective, it's not impossible
> >> > > totally.
> >> > >
> >> > > Thanks, Hongtao
> >> > >
> >> > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> >> > >
> >> > > > Hi dev team,
> >> > > >
> >> > > > I want to provide a message transmission future, which can be used
> >> for
> >> > > > message transmission in projects that already use skywalking.
> >> > > > This future of data transmission between multiple instances in the
> >> same
> >> > > > trace can be achieved by adding a new API of information
> >> transmission
> >> > in
> >> > > > each language level, which does not need a dependency on the
> >> existing
> >> > > > interface protocol.
> >> > > > For example, the execution process of a trace is A -> B -> C, Set
> >> > > transfer
> >> > > > content in A service, B and C services can obtain data through
> API.
> >> > > > This future is to use the message passing future of Skywaling,
> >> addition
> >> > > an
> >> > > > extension function. I'm not quite sure if it's appropriate to add
> >> it to
> >> > > > Skywalking?
> >> > > >
> >> > > > Currently, it can be applied to these scenarios:
> >> > > > 1. For the existing old system, It's not convenient to revise the
> >> data
> >> > of
> >> > > > the existing protocol. Through embedding transparent data,
> >> downstream
> >> > > > services can get data information.
> >> > > > 2. In the entry layer, such as zuul or other gateway services,
> >> > > > transparently transmits user identity or other customized
> >> information
> >> > to
> >> > > > the downstream services.
> >> > > >
> >> > > > Implementation plan:
> >> > > > 1. Store transmission data as Map in the current segment.
> >> > > > 2. Create a new protocol header for data transmission (JSON)
> >> > > > 3. When the downstream system receives the new protocol, it will
> >> > > transform
> >> > > > the data back to Map and store to the segment
> >> > > > 4. I think this implementation should apply to trace and ignored
> >> > segment,
> >> > > > otherwise, transmission data maybe will be lost.
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Hongtao Gao
> >> > >
> >> > > Apache SkyWalking && Apache ShardingSphere
> >> > > Twitter, @hanahmily
> >> > >
> >> >
> >>
> >
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Hi Han Liu
Positive feedback from Zipkin team, Read this from Adrian, Zipkin lead.
https://twitter.com/adrianfcole/status/1235424536608174080
Some similar features in the Zipkin Java SDK, brave project.
Sheng Wu 吴晟
Twitter, wusheng1108
Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> Hi Gao
>
> I am still confusing. If b3 used, then trace is the trace, there is
> nothing related to what we do.
> If application is instrumented by Zipkin SDK, then it is Zipkin's
> transmission capability, not us.
>
> In the proposal, it is clear that our agent is going to propagate things
> other injected and propagate them without touch. I can't see the link
> between this and your mesh case.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
>
>> Let me clarify what I'm talking about.
>>
>> In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
>> propagation, which means they can extract trace context from downstream
>> and
>> inject it to upstream.
>>
>> I want downside envoy sidecar instrumented by SkyWalking can wrap sw trace
>> context into B3 context to bypass target app/service to upside sidecar.
>>
>> When it happens, skywalking can trace application instrumented only by
>> Zipkin in service mesh scenario.
>>
>> I prefer to Zipkin that because it seems the actual standard tracing
>> protocol in the SM world. And we leverage it as a lower-level protocol to
>> figure out tracing context overlay.
>>
>>
>>
>>
>> On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com> wrote:
>>
>> > Hi Gao
>> >
>> > Look through your reply, are you talking about propagating zipkin
>> context
>> > or skywalking context? Your case is not clear to me.
>> >
>> > Sheng Wu 吴晟
>> > Twitter, wusheng1108
>> >
>> >
>> > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
>> >
>> > > Look good to me. I think this feature is also fantastic to the Service
>> > Mesh
>> > > scenario.
>> > > For now, OAP ingests send and receive messages from the sidecar and
>> uses
>> > > statistic analysis to link them to a topology. There's no trace here.
>> > >
>> > > If we implement the transmission protocol, the service workload can
>> > bypass
>> > > skywalking protocol, sidecar creates a segment and sinks it to OAP
>> > instead.
>> > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
>> > > connected by the application who implement Zipkin protocol. I know it
>> may
>> > > be
>> > > a big challenge for us, but from my perspective, it's not impossible
>> > > totally.
>> > >
>> > > Thanks, Hongtao
>> > >
>> > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
>> > >
>> > > > Hi dev team,
>> > > >
>> > > > I want to provide a message transmission future, which can be used
>> for
>> > > > message transmission in projects that already use skywalking.
>> > > > This future of data transmission between multiple instances in the
>> same
>> > > > trace can be achieved by adding a new API of information
>> transmission
>> > in
>> > > > each language level, which does not need a dependency on the
>> existing
>> > > > interface protocol.
>> > > > For example, the execution process of a trace is A -> B -> C, Set
>> > > transfer
>> > > > content in A service, B and C services can obtain data through API.
>> > > > This future is to use the message passing future of Skywaling,
>> addition
>> > > an
>> > > > extension function. I'm not quite sure if it's appropriate to add
>> it to
>> > > > Skywalking?
>> > > >
>> > > > Currently, it can be applied to these scenarios:
>> > > > 1. For the existing old system, It's not convenient to revise the
>> data
>> > of
>> > > > the existing protocol. Through embedding transparent data,
>> downstream
>> > > > services can get data information.
>> > > > 2. In the entry layer, such as zuul or other gateway services,
>> > > > transparently transmits user identity or other customized
>> information
>> > to
>> > > > the downstream services.
>> > > >
>> > > > Implementation plan:
>> > > > 1. Store transmission data as Map in the current segment.
>> > > > 2. Create a new protocol header for data transmission (JSON)
>> > > > 3. When the downstream system receives the new protocol, it will
>> > > transform
>> > > > the data back to Map and store to the segment
>> > > > 4. I think this implementation should apply to trace and ignored
>> > segment,
>> > > > otherwise, transmission data maybe will be lost.
>> > > >
>> > >
>> > >
>> > > --
>> > > Hongtao Gao
>> > >
>> > > Apache SkyWalking && Apache ShardingSphere
>> > > Twitter, @hanahmily
>> > >
>> >
>>
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Please send a PR for the protocol documentation first.
Once the team agrees on that, the implementation will be the easy part.
Sheng Wu 吴晟
Twitter, wusheng1108
han liu <mr...@gmail.com> 于2020年3月12日周四 下午2:07写道：
> Agree. I will submit a PR on the weekend to support the Java language, and
> write a document to explain the protocol so that the sniffer of other
> languages can be processed.
>
> Sheng Wu <wu...@gmail.com> 于2020年3月12日周四 下午1:38写道：
>
> > JSON works for me, but we need to make the documentation clear, we only
> > support string-key/string-value and one level JSON, in order to keep the
> > implementation of all language consistent same.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > han liu <mr...@gmail.com> 于2020年3月12日周四 下午12:32写道：
> >
> > > I see what you mean. Because we can't decide what users input, we may
> > make
> > > mistakes if we use the customized format directly.
> > > I think we can use JSON format to wrap all the data into a string, and
> > then
> > > Base64 encoding, transfer it in this way.
> > >
> > > Sheng Wu, ZhenXu Ke, what do you think?
> > >
> > > Sheng Wu <wu...@gmail.com>于2020年3月11日 周三上午8:54写道：
> > >
> > > > kezhenxu94 <ke...@163.com> 于2020年3月10日周二 下午6:31写道：
> > > >
> > > > > > 7. When transmitting, each user-defined key and value needs to be
> > > > base64
> > > > > >> encoded.
> > > > > >
> > > > > > `key:value` should be the internal format, I think we should make
> > > them
> > > > > > base64 encoding together, rather than putting the `:` in the
> > header.
> > > > >
> > > > > Then how do you determine which `:` is the delimiter of KV, if the
> > key
> > > > > contains `:`, since keys are user-generated, and UGC are always
> > > > > unpredictable
> > > > >
> > > >
> > > > I think that is protocol design part about how this format should
> look
> > > > like.
> > > > I could get your point about `:` could not be a regular string inside
> > the
> > > > package entity, agree.
> > > >
> > > > Han Liu
> > > > Please consider this in your protocol design.
> > > >
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > >
> >
>
Re: A new proposal of message transmission
Posted by han liu <mr...@gmail.com>.
Agree. I will submit a PR on the weekend to support the Java language, and
write a document to explain the protocol so that the sniffer of other
languages can be processed.
Sheng Wu <wu...@gmail.com> 于2020年3月12日周四 下午1:38写道：
> JSON works for me, but we need to make the documentation clear, we only
> support string-key/string-value and one level JSON, in order to keep the
> implementation of all language consistent same.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> han liu <mr...@gmail.com> 于2020年3月12日周四 下午12:32写道：
>
> > I see what you mean. Because we can't decide what users input, we may
> make
> > mistakes if we use the customized format directly.
> > I think we can use JSON format to wrap all the data into a string, and
> then
> > Base64 encoding, transfer it in this way.
> >
> > Sheng Wu, ZhenXu Ke, what do you think?
> >
> > Sheng Wu <wu...@gmail.com>于2020年3月11日 周三上午8:54写道：
> >
> > > kezhenxu94 <ke...@163.com> 于2020年3月10日周二 下午6:31写道：
> > >
> > > > > 7. When transmitting, each user-defined key and value needs to be
> > > base64
> > > > >> encoded.
> > > > >
> > > > > `key:value` should be the internal format, I think we should make
> > them
> > > > > base64 encoding together, rather than putting the `:` in the
> header.
> > > >
> > > > Then how do you determine which `:` is the delimiter of KV, if the
> key
> > > > contains `:`, since keys are user-generated, and UGC are always
> > > > unpredictable
> > > >
> > >
> > > I think that is protocol design part about how this format should look
> > > like.
> > > I could get your point about `:` could not be a regular string inside
> the
> > > package entity, agree.
> > >
> > > Han Liu
> > > Please consider this in your protocol design.
> > >
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> >
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
JSON works for me, but we need to make the documentation clear, we only
support string-key/string-value and one level JSON, in order to keep the
implementation of all language consistent same.
Sheng Wu 吴晟
Twitter, wusheng1108
han liu <mr...@gmail.com> 于2020年3月12日周四 下午12:32写道：
> I see what you mean. Because we can't decide what users input, we may make
> mistakes if we use the customized format directly.
> I think we can use JSON format to wrap all the data into a string, and then
> Base64 encoding, transfer it in this way.
>
> Sheng Wu, ZhenXu Ke, what do you think?
>
> Sheng Wu <wu...@gmail.com>于2020年3月11日 周三上午8:54写道：
>
> > kezhenxu94 <ke...@163.com> 于2020年3月10日周二 下午6:31写道：
> >
> > > > 7. When transmitting, each user-defined key and value needs to be
> > base64
> > > >> encoded.
> > > >
> > > > `key:value` should be the internal format, I think we should make
> them
> > > > base64 encoding together, rather than putting the `:` in the header.
> > >
> > > Then how do you determine which `:` is the delimiter of KV, if the key
> > > contains `:`, since keys are user-generated, and UGC are always
> > > unpredictable
> > >
> >
> > I think that is protocol design part about how this format should look
> > like.
> > I could get your point about `:` could not be a regular string inside the
> > package entity, agree.
> >
> > Han Liu
> > Please consider this in your protocol design.
> >
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
>
Re: A new proposal of message transmission
Posted by han liu <mr...@gmail.com>.
I see what you mean. Because we can't decide what users input, we may make
mistakes if we use the customized format directly.
I think we can use JSON format to wrap all the data into a string, and then
Base64 encoding, transfer it in this way.
Sheng Wu, ZhenXu Ke, what do you think?
Sheng Wu <wu...@gmail.com>于2020年3月11日 周三上午8:54写道：
> kezhenxu94 <ke...@163.com> 于2020年3月10日周二 下午6:31写道：
>
> > > 7. When transmitting, each user-defined key and value needs to be
> base64
> > >> encoded.
> > >
> > > `key:value` should be the internal format, I think we should make them
> > > base64 encoding together, rather than putting the `:` in the header.
> >
> > Then how do you determine which `:` is the delimiter of KV, if the key
> > contains `:`, since keys are user-generated, and UGC are always
> > unpredictable
> >
>
> I think that is protocol design part about how this format should look
> like.
> I could get your point about `:` could not be a regular string inside the
> package entity, agree.
>
> Han Liu
> Please consider this in your protocol design.
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
kezhenxu94 <ke...@163.com> 于2020年3月10日周二 下午6:31写道：
> > 7. When transmitting, each user-defined key and value needs to be base64
> >> encoded.
> >
> > `key:value` should be the internal format, I think we should make them
> > base64 encoding together, rather than putting the `:` in the header.
>
> Then how do you determine which `:` is the delimiter of KV, if the key
> contains `:`, since keys are user-generated, and UGC are always
> unpredictable
>
I think that is protocol design part about how this format should look like.
I could get your point about `:` could not be a regular string inside the
package entity, agree.
Han Liu
Please consider this in your protocol design.
Sheng Wu 吴晟
Twitter, wusheng1108
Re: A new proposal of message transmission
Posted by kezhenxu94 <ke...@163.com>.
> 7. When transmitting, each user-defined key and value needs to be base64
>> encoded.
>
> `key:value` should be the internal format, I think we should make them
> base64 encoding together, rather than putting the `:` in the header.
Then how do you determine which `:` is the delimiter of KV, if the key contains `:`, since keys are user-generated, and UGC are always unpredictable
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
han liu <mr...@gmail.com> 于2020年3月9日周一 下午5:06写道：
> Hi Sheng Wu, Here is my reply.
>
> Sheng Wu <wu...@gmail.com> 于2020年3月9日周一 下午3:43写道：
>
> > Reply inline.
> >
> >
> > han liu <mr...@gmail.com> 于2020年3月9日周一 下午3:30写道：
> >
> > > Hi dev team,
> > > I have designed some protocols for transparent transmission. Please
> help
> > me
> > > to see what is missing or what is wrong.
> > >
> > > 1. Provide API-level data transmission function, use get or put
> functions
> > > to operate data, they are similar to Map in java.
> > > 2. Each data is in the form of key-value, and both key and value are
> > string
> > > types.
> > > 3. Multiple identical keys will be overwritten and the original old
> data
> > > will be returned to the user.
> > > 4. Supports up to 3 key definitions, and the length of the value in
> each
> > > key is 128 bytes. Logging occurs when the size limit is exceeded.
> > >
> >
> > Usually, agent doesn't log in the status. You could consider adding this
> to
> > the return value. The log is expensive.
> >
>
> Sure, I think it is possible to use a return data structure similar to
> create the task in the profile[1]. Use "error reason" and "old data" to
> return.
>
> >
> >
> > > 5. The size limit supports modification at the agent configuration
> level.
> > > 6. Define a new header with key "sw7-correlation" and value in the
> format
> > > "key1:value1|key2:value2".
> >
> > 7. When transmitting, each user-defined key and value needs to be base64
> > > encoded.
> > >
> >
> > `key:value` should be the internal format, I think we should make them
> > base64 encoding together, rather than putting the `:` in the header.
> >
>
> I think this is a good idea. Change it to the code formate like this:
> "base64(key:value|key:value)"
>
> >
> >
> > > 8. Data transmission must be performed in the Tracing environment.
> > > 9. Data is stored in TracingContext or IgnoreTraceContext.
> > >
> >
> > This is for Java only, Other language agents could be flexible based on
> > their own.
> > This should not represent in the official final protocol doc.
> >
>
> Yes, but I think that 2, 6, 7, 8, 9 should be common across languages.
>
The language API style could be a recommendation, not a requirement in the
protocol.
If we could keep them in the same, then it will be convenient for the
end-users.
Sheng Wu 吴晟
Twitter, wusheng1108
>
> [1]
>
> https://github.com/apache/skywalking-query-protocol/blob/06373bf0d1204b6d3f410c0daf4c7f72529f28a0/profile.graphqls#L36-L41
>
>
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > > 10. Copy the currently existing data when taking a snapshot, and put
> the
> > > data into a new context when continuing.
> > >
> > > Sheng Wu <wu...@gmail.com> 于2020年3月6日周五 下午4:51写道：
> > >
> > > > Update from off list talk.
> > > >
> > > > Short version, Gao misunderstood this proposal. There is nothing to
> be
> > > > added into this proposal.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午11:18写道：
> > > >
> > > > > Sure, will call you at tomorrow 10am UTC+8.
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
> > > > >
> > > > >> Hi Sheng
> > > > >> I prefer to schedule a hangout(maybe tomorrow) to discuss the
> > details
> > > of
> > > > >> my
> > > > >> proposal, which seems not suitable to dig into it in this thread.
> > This
> > > > >> proposal inspires me to find out a new approach to propagate sw
> > > tracing
> > > > >> context.
> > > > >>
> > > > >> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> > > > >>
> > > > >> > Hi Gao
> > > > >> >
> > > > >> > I am still confusing. If b3 used, then trace is the trace, there
> > is
> > > > >> nothing
> > > > >> > related to what we do.
> > > > >> > If application is instrumented by Zipkin SDK, then it is
> Zipkin's
> > > > >> > transmission capability, not us.
> > > > >> >
> > > > >> > In the proposal, it is clear that our agent is going to
> propagate
> > > > things
> > > > >> > other injected and propagate them without touch. I can't see the
> > > link
> > > > >> > between this and your mesh case.
> > > > >> >
> > > > >> > Sheng Wu 吴晟
> > > > >> > Twitter, wusheng1108
> > > > >> >
> > > > >> >
> > > > >> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> > > > >> >
> > > > >> > > Let me clarify what I'm talking about.
> > > > >> > >
> > > > >> > > In the Istio demo, just like Bookinfo, all of the apps
> implement
> > > > >> Zipkin
> > > > >> > > propagation, which means they can extract trace context from
> > > > >> downstream
> > > > >> > and
> > > > >> > > inject it to upstream.
> > > > >> > >
> > > > >> > > I want downside envoy sidecar instrumented by SkyWalking can
> > wrap
> > > sw
> > > > >> > trace
> > > > >> > > context into B3 context to bypass target app/service to upside
> > > > >> sidecar.
> > > > >> > >
> > > > >> > > When it happens, skywalking can trace application instrumented
> > > only
> > > > by
> > > > >> > > Zipkin in service mesh scenario.
> > > > >> > >
> > > > >> > > I prefer to Zipkin that because it seems the actual standard
> > > tracing
> > > > >> > > protocol in the SM world. And we leverage it as a lower-level
> > > > >> protocol to
> > > > >> > > figure out tracing context overlay.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <
> > wu.sheng.841108@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Hi Gao
> > > > >> > > >
> > > > >> > > > Look through your reply, are you talking about propagating
> > > zipkin
> > > > >> > context
> > > > >> > > > or skywalking context? Your case is not clear to me.
> > > > >> > > >
> > > > >> > > > Sheng Wu 吴晟
> > > > >> > > > Twitter, wusheng1108
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > > > >> > > >
> > > > >> > > > > Look good to me. I think this feature is also fantastic to
> > the
> > > > >> > Service
> > > > >> > > > Mesh
> > > > >> > > > > scenario.
> > > > >> > > > > For now, OAP ingests send and receive messages from the
> > > sidecar
> > > > >> and
> > > > >> > > uses
> > > > >> > > > > statistic analysis to link them to a topology. There's no
> > > trace
> > > > >> here.
> > > > >> > > > >
> > > > >> > > > > If we implement the transmission protocol, the service
> > > workload
> > > > >> can
> > > > >> > > > bypass
> > > > >> > > > > skywalking protocol, sidecar creates a segment and sinks
> it
> > to
> > > > OAP
> > > > >> > > > instead.
> > > > >> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our
> > > trace
> > > > >> is
> > > > >> > > > > connected by the application who implement Zipkin
> protocol.
> > I
> > > > >> know it
> > > > >> > > may
> > > > >> > > > > be
> > > > >> > > > > a big challenge for us, but from my perspective, it's not
> > > > >> impossible
> > > > >> > > > > totally.
> > > > >> > > > >
> > > > >> > > > > Thanks, Hongtao
> > > > >> > > > >
> > > > >> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > > > >> > > > >
> > > > >> > > > > > Hi dev team,
> > > > >> > > > > >
> > > > >> > > > > > I want to provide a message transmission future, which
> can
> > > be
> > > > >> used
> > > > >> > > for
> > > > >> > > > > > message transmission in projects that already use
> > > skywalking.
> > > > >> > > > > > This future of data transmission between multiple
> > instances
> > > in
> > > > >> the
> > > > >> > > same
> > > > >> > > > > > trace can be achieved by adding a new API of information
> > > > >> > transmission
> > > > >> > > > in
> > > > >> > > > > > each language level, which does not need a dependency on
> > the
> > > > >> > existing
> > > > >> > > > > > interface protocol.
> > > > >> > > > > > For example, the execution process of a trace is A -> B
> ->
> > > C,
> > > > >> Set
> > > > >> > > > > transfer
> > > > >> > > > > > content in A service, B and C services can obtain data
> > > through
> > > > >> API.
> > > > >> > > > > > This future is to use the message passing future of
> > > Skywaling,
> > > > >> > > addition
> > > > >> > > > > an
> > > > >> > > > > > extension function. I'm not quite sure if it's
> appropriate
> > > to
> > > > >> add
> > > > >> > it
> > > > >> > > to
> > > > >> > > > > > Skywalking?
> > > > >> > > > > >
> > > > >> > > > > > Currently, it can be applied to these scenarios:
> > > > >> > > > > > 1. For the existing old system, It's not convenient to
> > > revise
> > > > >> the
> > > > >> > > data
> > > > >> > > > of
> > > > >> > > > > > the existing protocol. Through embedding transparent
> data,
> > > > >> > downstream
> > > > >> > > > > > services can get data information.
> > > > >> > > > > > 2. In the entry layer, such as zuul or other gateway
> > > services,
> > > > >> > > > > > transparently transmits user identity or other
> customized
> > > > >> > information
> > > > >> > > > to
> > > > >> > > > > > the downstream services.
> > > > >> > > > > >
> > > > >> > > > > > Implementation plan:
> > > > >> > > > > > 1. Store transmission data as Map in the current
> segment.
> > > > >> > > > > > 2. Create a new protocol header for data transmission
> > (JSON)
> > > > >> > > > > > 3. When the downstream system receives the new protocol,
> > it
> > > > will
> > > > >> > > > > transform
> > > > >> > > > > > the data back to Map and store to the segment
> > > > >> > > > > > 4. I think this implementation should apply to trace and
> > > > ignored
> > > > >> > > > segment,
> > > > >> > > > > > otherwise, transmission data maybe will be lost.
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > > Hongtao Gao
> > > > >> > > > >
> > > > >> > > > > Apache SkyWalking && Apache ShardingSphere
> > > > >> > > > > Twitter, @hanahmily
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Hongtao Gao
> > > > >>
> > > > >> Apache SkyWalking && Apache ShardingSphere
> > > > >> Twitter, @hanahmily
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: A new proposal of message transmission
Posted by han liu <mr...@gmail.com>.
Hi Sheng Wu, Here is my reply.
Sheng Wu <wu...@gmail.com> 于2020年3月9日周一 下午3:43写道：
> Reply inline.
>
>
> han liu <mr...@gmail.com> 于2020年3月9日周一 下午3:30写道：
>
> > Hi dev team,
> > I have designed some protocols for transparent transmission. Please help
> me
> > to see what is missing or what is wrong.
> >
> > 1. Provide API-level data transmission function, use get or put functions
> > to operate data, they are similar to Map in java.
> > 2. Each data is in the form of key-value, and both key and value are
> string
> > types.
> > 3. Multiple identical keys will be overwritten and the original old data
> > will be returned to the user.
> > 4. Supports up to 3 key definitions, and the length of the value in each
> > key is 128 bytes. Logging occurs when the size limit is exceeded.
> >
>
> Usually, agent doesn't log in the status. You could consider adding this to
> the return value. The log is expensive.
>
Sure, I think it is possible to use a return data structure similar to
create the task in the profile[1]. Use "error reason" and "old data" to
return.
>
>
> > 5. The size limit supports modification at the agent configuration level.
> > 6. Define a new header with key "sw7-correlation" and value in the format
> > "key1:value1|key2:value2".
>
> 7. When transmitting, each user-defined key and value needs to be base64
> > encoded.
> >
>
> `key:value` should be the internal format, I think we should make them
> base64 encoding together, rather than putting the `:` in the header.
>
I think this is a good idea. Change it to the code formate like this:
"base64(key:value|key:value)"
>
>
> > 8. Data transmission must be performed in the Tracing environment.
> > 9. Data is stored in TracingContext or IgnoreTraceContext.
> >
>
> This is for Java only, Other language agents could be flexible based on
> their own.
> This should not represent in the official final protocol doc.
>
Yes, but I think that 2, 6, 7, 8, 9 should be common across languages.
[1]
https://github.com/apache/skywalking-query-protocol/blob/06373bf0d1204b6d3f410c0daf4c7f72529f28a0/profile.graphqls#L36-L41
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> > 10. Copy the currently existing data when taking a snapshot, and put the
> > data into a new context when continuing.
> >
> > Sheng Wu <wu...@gmail.com> 于2020年3月6日周五 下午4:51写道：
> >
> > > Update from off list talk.
> > >
> > > Short version, Gao misunderstood this proposal. There is nothing to be
> > > added into this proposal.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午11:18写道：
> > >
> > > > Sure, will call you at tomorrow 10am UTC+8.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
> > > >
> > > >> Hi Sheng
> > > >> I prefer to schedule a hangout(maybe tomorrow) to discuss the
> details
> > of
> > > >> my
> > > >> proposal, which seems not suitable to dig into it in this thread.
> This
> > > >> proposal inspires me to find out a new approach to propagate sw
> > tracing
> > > >> context.
> > > >>
> > > >> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> > > >>
> > > >> > Hi Gao
> > > >> >
> > > >> > I am still confusing. If b3 used, then trace is the trace, there
> is
> > > >> nothing
> > > >> > related to what we do.
> > > >> > If application is instrumented by Zipkin SDK, then it is Zipkin's
> > > >> > transmission capability, not us.
> > > >> >
> > > >> > In the proposal, it is clear that our agent is going to propagate
> > > things
> > > >> > other injected and propagate them without touch. I can't see the
> > link
> > > >> > between this and your mesh case.
> > > >> >
> > > >> > Sheng Wu 吴晟
> > > >> > Twitter, wusheng1108
> > > >> >
> > > >> >
> > > >> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> > > >> >
> > > >> > > Let me clarify what I'm talking about.
> > > >> > >
> > > >> > > In the Istio demo, just like Bookinfo, all of the apps implement
> > > >> Zipkin
> > > >> > > propagation, which means they can extract trace context from
> > > >> downstream
> > > >> > and
> > > >> > > inject it to upstream.
> > > >> > >
> > > >> > > I want downside envoy sidecar instrumented by SkyWalking can
> wrap
> > sw
> > > >> > trace
> > > >> > > context into B3 context to bypass target app/service to upside
> > > >> sidecar.
> > > >> > >
> > > >> > > When it happens, skywalking can trace application instrumented
> > only
> > > by
> > > >> > > Zipkin in service mesh scenario.
> > > >> > >
> > > >> > > I prefer to Zipkin that because it seems the actual standard
> > tracing
> > > >> > > protocol in the SM world. And we leverage it as a lower-level
> > > >> protocol to
> > > >> > > figure out tracing context overlay.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <
> wu.sheng.841108@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > Hi Gao
> > > >> > > >
> > > >> > > > Look through your reply, are you talking about propagating
> > zipkin
> > > >> > context
> > > >> > > > or skywalking context? Your case is not clear to me.
> > > >> > > >
> > > >> > > > Sheng Wu 吴晟
> > > >> > > > Twitter, wusheng1108
> > > >> > > >
> > > >> > > >
> > > >> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > > >> > > >
> > > >> > > > > Look good to me. I think this feature is also fantastic to
> the
> > > >> > Service
> > > >> > > > Mesh
> > > >> > > > > scenario.
> > > >> > > > > For now, OAP ingests send and receive messages from the
> > sidecar
> > > >> and
> > > >> > > uses
> > > >> > > > > statistic analysis to link them to a topology. There's no
> > trace
> > > >> here.
> > > >> > > > >
> > > >> > > > > If we implement the transmission protocol, the service
> > workload
> > > >> can
> > > >> > > > bypass
> > > >> > > > > skywalking protocol, sidecar creates a segment and sinks it
> to
> > > OAP
> > > >> > > > instead.
> > > >> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our
> > trace
> > > >> is
> > > >> > > > > connected by the application who implement Zipkin protocol.
> I
> > > >> know it
> > > >> > > may
> > > >> > > > > be
> > > >> > > > > a big challenge for us, but from my perspective, it's not
> > > >> impossible
> > > >> > > > > totally.
> > > >> > > > >
> > > >> > > > > Thanks, Hongtao
> > > >> > > > >
> > > >> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > > >> > > > >
> > > >> > > > > > Hi dev team,
> > > >> > > > > >
> > > >> > > > > > I want to provide a message transmission future, which can
> > be
> > > >> used
> > > >> > > for
> > > >> > > > > > message transmission in projects that already use
> > skywalking.
> > > >> > > > > > This future of data transmission between multiple
> instances
> > in
> > > >> the
> > > >> > > same
> > > >> > > > > > trace can be achieved by adding a new API of information
> > > >> > transmission
> > > >> > > > in
> > > >> > > > > > each language level, which does not need a dependency on
> the
> > > >> > existing
> > > >> > > > > > interface protocol.
> > > >> > > > > > For example, the execution process of a trace is A -> B ->
> > C,
> > > >> Set
> > > >> > > > > transfer
> > > >> > > > > > content in A service, B and C services can obtain data
> > through
> > > >> API.
> > > >> > > > > > This future is to use the message passing future of
> > Skywaling,
> > > >> > > addition
> > > >> > > > > an
> > > >> > > > > > extension function. I'm not quite sure if it's appropriate
> > to
> > > >> add
> > > >> > it
> > > >> > > to
> > > >> > > > > > Skywalking?
> > > >> > > > > >
> > > >> > > > > > Currently, it can be applied to these scenarios:
> > > >> > > > > > 1. For the existing old system, It's not convenient to
> > revise
> > > >> the
> > > >> > > data
> > > >> > > > of
> > > >> > > > > > the existing protocol. Through embedding transparent data,
> > > >> > downstream
> > > >> > > > > > services can get data information.
> > > >> > > > > > 2. In the entry layer, such as zuul or other gateway
> > services,
> > > >> > > > > > transparently transmits user identity or other customized
> > > >> > information
> > > >> > > > to
> > > >> > > > > > the downstream services.
> > > >> > > > > >
> > > >> > > > > > Implementation plan:
> > > >> > > > > > 1. Store transmission data as Map in the current segment.
> > > >> > > > > > 2. Create a new protocol header for data transmission
> (JSON)
> > > >> > > > > > 3. When the downstream system receives the new protocol,
> it
> > > will
> > > >> > > > > transform
> > > >> > > > > > the data back to Map and store to the segment
> > > >> > > > > > 4. I think this implementation should apply to trace and
> > > ignored
> > > >> > > > segment,
> > > >> > > > > > otherwise, transmission data maybe will be lost.
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > Hongtao Gao
> > > >> > > > >
> > > >> > > > > Apache SkyWalking && Apache ShardingSphere
> > > >> > > > > Twitter, @hanahmily
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Hongtao Gao
> > > >>
> > > >> Apache SkyWalking && Apache ShardingSphere
> > > >> Twitter, @hanahmily
> > > >>
> > > >
> > >
> >
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Reply inline.
han liu <mr...@gmail.com> 于2020年3月9日周一 下午3:30写道：
> Hi dev team,
> I have designed some protocols for transparent transmission. Please help me
> to see what is missing or what is wrong.
>
> 1. Provide API-level data transmission function, use get or put functions
> to operate data, they are similar to Map in java.
> 2. Each data is in the form of key-value, and both key and value are string
> types.
> 3. Multiple identical keys will be overwritten and the original old data
> will be returned to the user.
> 4. Supports up to 3 key definitions, and the length of the value in each
> key is 128 bytes. Logging occurs when the size limit is exceeded.
>
Usually, agent doesn't log in the status. You could consider adding this to
the return value. The log is expensive.
> 5. The size limit supports modification at the agent configuration level.
> 6. Define a new header with key "sw7-correlation" and value in the format
> "key1:value1|key2:value2".
7. When transmitting, each user-defined key and value needs to be base64
> encoded.
>
`key:value` should be the internal format, I think we should make them
base64 encoding together, rather than putting the `:` in the header.
> 8. Data transmission must be performed in the Tracing environment.
> 9. Data is stored in TracingContext or IgnoreTraceContext.
>
This is for Java only, Other language agents could be flexible based on
their own.
This should not represent in the official final protocol doc.
Sheng Wu 吴晟
Twitter, wusheng1108
> 10. Copy the currently existing data when taking a snapshot, and put the
> data into a new context when continuing.
>
> Sheng Wu <wu...@gmail.com> 于2020年3月6日周五 下午4:51写道：
>
> > Update from off list talk.
> >
> > Short version, Gao misunderstood this proposal. There is nothing to be
> > added into this proposal.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午11:18写道：
> >
> > > Sure, will call you at tomorrow 10am UTC+8.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
> > >
> > >> Hi Sheng
> > >> I prefer to schedule a hangout(maybe tomorrow) to discuss the details
> of
> > >> my
> > >> proposal, which seems not suitable to dig into it in this thread. This
> > >> proposal inspires me to find out a new approach to propagate sw
> tracing
> > >> context.
> > >>
> > >> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> > >>
> > >> > Hi Gao
> > >> >
> > >> > I am still confusing. If b3 used, then trace is the trace, there is
> > >> nothing
> > >> > related to what we do.
> > >> > If application is instrumented by Zipkin SDK, then it is Zipkin's
> > >> > transmission capability, not us.
> > >> >
> > >> > In the proposal, it is clear that our agent is going to propagate
> > things
> > >> > other injected and propagate them without touch. I can't see the
> link
> > >> > between this and your mesh case.
> > >> >
> > >> > Sheng Wu 吴晟
> > >> > Twitter, wusheng1108
> > >> >
> > >> >
> > >> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> > >> >
> > >> > > Let me clarify what I'm talking about.
> > >> > >
> > >> > > In the Istio demo, just like Bookinfo, all of the apps implement
> > >> Zipkin
> > >> > > propagation, which means they can extract trace context from
> > >> downstream
> > >> > and
> > >> > > inject it to upstream.
> > >> > >
> > >> > > I want downside envoy sidecar instrumented by SkyWalking can wrap
> sw
> > >> > trace
> > >> > > context into B3 context to bypass target app/service to upside
> > >> sidecar.
> > >> > >
> > >> > > When it happens, skywalking can trace application instrumented
> only
> > by
> > >> > > Zipkin in service mesh scenario.
> > >> > >
> > >> > > I prefer to Zipkin that because it seems the actual standard
> tracing
> > >> > > protocol in the SM world. And we leverage it as a lower-level
> > >> protocol to
> > >> > > figure out tracing context overlay.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu.sheng.841108@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > Hi Gao
> > >> > > >
> > >> > > > Look through your reply, are you talking about propagating
> zipkin
> > >> > context
> > >> > > > or skywalking context? Your case is not clear to me.
> > >> > > >
> > >> > > > Sheng Wu 吴晟
> > >> > > > Twitter, wusheng1108
> > >> > > >
> > >> > > >
> > >> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > >> > > >
> > >> > > > > Look good to me. I think this feature is also fantastic to the
> > >> > Service
> > >> > > > Mesh
> > >> > > > > scenario.
> > >> > > > > For now, OAP ingests send and receive messages from the
> sidecar
> > >> and
> > >> > > uses
> > >> > > > > statistic analysis to link them to a topology. There's no
> trace
> > >> here.
> > >> > > > >
> > >> > > > > If we implement the transmission protocol, the service
> workload
> > >> can
> > >> > > > bypass
> > >> > > > > skywalking protocol, sidecar creates a segment and sinks it to
> > OAP
> > >> > > > instead.
> > >> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our
> trace
> > >> is
> > >> > > > > connected by the application who implement Zipkin protocol. I
> > >> know it
> > >> > > may
> > >> > > > > be
> > >> > > > > a big challenge for us, but from my perspective, it's not
> > >> impossible
> > >> > > > > totally.
> > >> > > > >
> > >> > > > > Thanks, Hongtao
> > >> > > > >
> > >> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > >> > > > >
> > >> > > > > > Hi dev team,
> > >> > > > > >
> > >> > > > > > I want to provide a message transmission future, which can
> be
> > >> used
> > >> > > for
> > >> > > > > > message transmission in projects that already use
> skywalking.
> > >> > > > > > This future of data transmission between multiple instances
> in
> > >> the
> > >> > > same
> > >> > > > > > trace can be achieved by adding a new API of information
> > >> > transmission
> > >> > > > in
> > >> > > > > > each language level, which does not need a dependency on the
> > >> > existing
> > >> > > > > > interface protocol.
> > >> > > > > > For example, the execution process of a trace is A -> B ->
> C,
> > >> Set
> > >> > > > > transfer
> > >> > > > > > content in A service, B and C services can obtain data
> through
> > >> API.
> > >> > > > > > This future is to use the message passing future of
> Skywaling,
> > >> > > addition
> > >> > > > > an
> > >> > > > > > extension function. I'm not quite sure if it's appropriate
> to
> > >> add
> > >> > it
> > >> > > to
> > >> > > > > > Skywalking?
> > >> > > > > >
> > >> > > > > > Currently, it can be applied to these scenarios:
> > >> > > > > > 1. For the existing old system, It's not convenient to
> revise
> > >> the
> > >> > > data
> > >> > > > of
> > >> > > > > > the existing protocol. Through embedding transparent data,
> > >> > downstream
> > >> > > > > > services can get data information.
> > >> > > > > > 2. In the entry layer, such as zuul or other gateway
> services,
> > >> > > > > > transparently transmits user identity or other customized
> > >> > information
> > >> > > > to
> > >> > > > > > the downstream services.
> > >> > > > > >
> > >> > > > > > Implementation plan:
> > >> > > > > > 1. Store transmission data as Map in the current segment.
> > >> > > > > > 2. Create a new protocol header for data transmission (JSON)
> > >> > > > > > 3. When the downstream system receives the new protocol, it
> > will
> > >> > > > > transform
> > >> > > > > > the data back to Map and store to the segment
> > >> > > > > > 4. I think this implementation should apply to trace and
> > ignored
> > >> > > > segment,
> > >> > > > > > otherwise, transmission data maybe will be lost.
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Hongtao Gao
> > >> > > > >
> > >> > > > > Apache SkyWalking && Apache ShardingSphere
> > >> > > > > Twitter, @hanahmily
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Hongtao Gao
> > >>
> > >> Apache SkyWalking && Apache ShardingSphere
> > >> Twitter, @hanahmily
> > >>
> > >
> >
>
Re: A new proposal of message transmission
Posted by han liu <mr...@gmail.com>.
Hi dev team,
I have designed some protocols for transparent transmission. Please help me
to see what is missing or what is wrong.
1. Provide API-level data transmission function, use get or put functions
to operate data, they are similar to Map in java.
2. Each data is in the form of key-value, and both key and value are string
types.
3. Multiple identical keys will be overwritten and the original old data
will be returned to the user.
4. Supports up to 3 key definitions, and the length of the value in each
key is 128 bytes. Logging occurs when the size limit is exceeded.
5. The size limit supports modification at the agent configuration level.
6. Define a new header with key "sw7-correlation" and value in the format
"key1:value1|key2:value2".
7. When transmitting, each user-defined key and value needs to be base64
encoded.
8. Data transmission must be performed in the Tracing environment.
9. Data is stored in TracingContext or IgnoreTraceContext.
10. Copy the currently existing data when taking a snapshot, and put the
data into a new context when continuing.
Sheng Wu <wu...@gmail.com> 于2020年3月6日周五 下午4:51写道：
> Update from off list talk.
>
> Short version, Gao misunderstood this proposal. There is nothing to be
> added into this proposal.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午11:18写道：
>
> > Sure, will call you at tomorrow 10am UTC+8.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
> >
> >> Hi Sheng
> >> I prefer to schedule a hangout(maybe tomorrow) to discuss the details of
> >> my
> >> proposal, which seems not suitable to dig into it in this thread. This
> >> proposal inspires me to find out a new approach to propagate sw tracing
> >> context.
> >>
> >> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> >>
> >> > Hi Gao
> >> >
> >> > I am still confusing. If b3 used, then trace is the trace, there is
> >> nothing
> >> > related to what we do.
> >> > If application is instrumented by Zipkin SDK, then it is Zipkin's
> >> > transmission capability, not us.
> >> >
> >> > In the proposal, it is clear that our agent is going to propagate
> things
> >> > other injected and propagate them without touch. I can't see the link
> >> > between this and your mesh case.
> >> >
> >> > Sheng Wu 吴晟
> >> > Twitter, wusheng1108
> >> >
> >> >
> >> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> >> >
> >> > > Let me clarify what I'm talking about.
> >> > >
> >> > > In the Istio demo, just like Bookinfo, all of the apps implement
> >> Zipkin
> >> > > propagation, which means they can extract trace context from
> >> downstream
> >> > and
> >> > > inject it to upstream.
> >> > >
> >> > > I want downside envoy sidecar instrumented by SkyWalking can wrap sw
> >> > trace
> >> > > context into B3 context to bypass target app/service to upside
> >> sidecar.
> >> > >
> >> > > When it happens, skywalking can trace application instrumented only
> by
> >> > > Zipkin in service mesh scenario.
> >> > >
> >> > > I prefer to Zipkin that because it seems the actual standard tracing
> >> > > protocol in the SM world. And we leverage it as a lower-level
> >> protocol to
> >> > > figure out tracing context overlay.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Hi Gao
> >> > > >
> >> > > > Look through your reply, are you talking about propagating zipkin
> >> > context
> >> > > > or skywalking context? Your case is not clear to me.
> >> > > >
> >> > > > Sheng Wu 吴晟
> >> > > > Twitter, wusheng1108
> >> > > >
> >> > > >
> >> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> >> > > >
> >> > > > > Look good to me. I think this feature is also fantastic to the
> >> > Service
> >> > > > Mesh
> >> > > > > scenario.
> >> > > > > For now, OAP ingests send and receive messages from the sidecar
> >> and
> >> > > uses
> >> > > > > statistic analysis to link them to a topology. There's no trace
> >> here.
> >> > > > >
> >> > > > > If we implement the transmission protocol, the service workload
> >> can
> >> > > > bypass
> >> > > > > skywalking protocol, sidecar creates a segment and sinks it to
> OAP
> >> > > > instead.
> >> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace
> >> is
> >> > > > > connected by the application who implement Zipkin protocol. I
> >> know it
> >> > > may
> >> > > > > be
> >> > > > > a big challenge for us, but from my perspective, it's not
> >> impossible
> >> > > > > totally.
> >> > > > >
> >> > > > > Thanks, Hongtao
> >> > > > >
> >> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> >> > > > >
> >> > > > > > Hi dev team,
> >> > > > > >
> >> > > > > > I want to provide a message transmission future, which can be
> >> used
> >> > > for
> >> > > > > > message transmission in projects that already use skywalking.
> >> > > > > > This future of data transmission between multiple instances in
> >> the
> >> > > same
> >> > > > > > trace can be achieved by adding a new API of information
> >> > transmission
> >> > > > in
> >> > > > > > each language level, which does not need a dependency on the
> >> > existing
> >> > > > > > interface protocol.
> >> > > > > > For example, the execution process of a trace is A -> B -> C,
> >> Set
> >> > > > > transfer
> >> > > > > > content in A service, B and C services can obtain data through
> >> API.
> >> > > > > > This future is to use the message passing future of Skywaling,
> >> > > addition
> >> > > > > an
> >> > > > > > extension function. I'm not quite sure if it's appropriate to
> >> add
> >> > it
> >> > > to
> >> > > > > > Skywalking?
> >> > > > > >
> >> > > > > > Currently, it can be applied to these scenarios:
> >> > > > > > 1. For the existing old system, It's not convenient to revise
> >> the
> >> > > data
> >> > > > of
> >> > > > > > the existing protocol. Through embedding transparent data,
> >> > downstream
> >> > > > > > services can get data information.
> >> > > > > > 2. In the entry layer, such as zuul or other gateway services,
> >> > > > > > transparently transmits user identity or other customized
> >> > information
> >> > > > to
> >> > > > > > the downstream services.
> >> > > > > >
> >> > > > > > Implementation plan:
> >> > > > > > 1. Store transmission data as Map in the current segment.
> >> > > > > > 2. Create a new protocol header for data transmission (JSON)
> >> > > > > > 3. When the downstream system receives the new protocol, it
> will
> >> > > > > transform
> >> > > > > > the data back to Map and store to the segment
> >> > > > > > 4. I think this implementation should apply to trace and
> ignored
> >> > > > segment,
> >> > > > > > otherwise, transmission data maybe will be lost.
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Hongtao Gao
> >> > > > >
> >> > > > > Apache SkyWalking && Apache ShardingSphere
> >> > > > > Twitter, @hanahmily
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Hongtao Gao
> >>
> >> Apache SkyWalking && Apache ShardingSphere
> >> Twitter, @hanahmily
> >>
> >
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Update from off list talk.
Short version, Gao misunderstood this proposal. There is nothing to be
added into this proposal.
Sheng Wu 吴晟
Twitter, wusheng1108
Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 下午11:18写道：
> Sure, will call you at tomorrow 10am UTC+8.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
>
>> Hi Sheng
>> I prefer to schedule a hangout(maybe tomorrow) to discuss the details of
>> my
>> proposal, which seems not suitable to dig into it in this thread. This
>> proposal inspires me to find out a new approach to propagate sw tracing
>> context.
>>
>> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
>>
>> > Hi Gao
>> >
>> > I am still confusing. If b3 used, then trace is the trace, there is
>> nothing
>> > related to what we do.
>> > If application is instrumented by Zipkin SDK, then it is Zipkin's
>> > transmission capability, not us.
>> >
>> > In the proposal, it is clear that our agent is going to propagate things
>> > other injected and propagate them without touch. I can't see the link
>> > between this and your mesh case.
>> >
>> > Sheng Wu 吴晟
>> > Twitter, wusheng1108
>> >
>> >
>> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
>> >
>> > > Let me clarify what I'm talking about.
>> > >
>> > > In the Istio demo, just like Bookinfo, all of the apps implement
>> Zipkin
>> > > propagation, which means they can extract trace context from
>> downstream
>> > and
>> > > inject it to upstream.
>> > >
>> > > I want downside envoy sidecar instrumented by SkyWalking can wrap sw
>> > trace
>> > > context into B3 context to bypass target app/service to upside
>> sidecar.
>> > >
>> > > When it happens, skywalking can trace application instrumented only by
>> > > Zipkin in service mesh scenario.
>> > >
>> > > I prefer to Zipkin that because it seems the actual standard tracing
>> > > protocol in the SM world. And we leverage it as a lower-level
>> protocol to
>> > > figure out tracing context overlay.
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Gao
>> > > >
>> > > > Look through your reply, are you talking about propagating zipkin
>> > context
>> > > > or skywalking context? Your case is not clear to me.
>> > > >
>> > > > Sheng Wu 吴晟
>> > > > Twitter, wusheng1108
>> > > >
>> > > >
>> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
>> > > >
>> > > > > Look good to me. I think this feature is also fantastic to the
>> > Service
>> > > > Mesh
>> > > > > scenario.
>> > > > > For now, OAP ingests send and receive messages from the sidecar
>> and
>> > > uses
>> > > > > statistic analysis to link them to a topology. There's no trace
>> here.
>> > > > >
>> > > > > If we implement the transmission protocol, the service workload
>> can
>> > > > bypass
>> > > > > skywalking protocol, sidecar creates a segment and sinks it to OAP
>> > > > instead.
>> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace
>> is
>> > > > > connected by the application who implement Zipkin protocol. I
>> know it
>> > > may
>> > > > > be
>> > > > > a big challenge for us, but from my perspective, it's not
>> impossible
>> > > > > totally.
>> > > > >
>> > > > > Thanks, Hongtao
>> > > > >
>> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
>> > > > >
>> > > > > > Hi dev team,
>> > > > > >
>> > > > > > I want to provide a message transmission future, which can be
>> used
>> > > for
>> > > > > > message transmission in projects that already use skywalking.
>> > > > > > This future of data transmission between multiple instances in
>> the
>> > > same
>> > > > > > trace can be achieved by adding a new API of information
>> > transmission
>> > > > in
>> > > > > > each language level, which does not need a dependency on the
>> > existing
>> > > > > > interface protocol.
>> > > > > > For example, the execution process of a trace is A -> B -> C,
>> Set
>> > > > > transfer
>> > > > > > content in A service, B and C services can obtain data through
>> API.
>> > > > > > This future is to use the message passing future of Skywaling,
>> > > addition
>> > > > > an
>> > > > > > extension function. I'm not quite sure if it's appropriate to
>> add
>> > it
>> > > to
>> > > > > > Skywalking?
>> > > > > >
>> > > > > > Currently, it can be applied to these scenarios:
>> > > > > > 1. For the existing old system, It's not convenient to revise
>> the
>> > > data
>> > > > of
>> > > > > > the existing protocol. Through embedding transparent data,
>> > downstream
>> > > > > > services can get data information.
>> > > > > > 2. In the entry layer, such as zuul or other gateway services,
>> > > > > > transparently transmits user identity or other customized
>> > information
>> > > > to
>> > > > > > the downstream services.
>> > > > > >
>> > > > > > Implementation plan:
>> > > > > > 1. Store transmission data as Map in the current segment.
>> > > > > > 2. Create a new protocol header for data transmission (JSON)
>> > > > > > 3. When the downstream system receives the new protocol, it will
>> > > > > transform
>> > > > > > the data back to Map and store to the segment
>> > > > > > 4. I think this implementation should apply to trace and ignored
>> > > > segment,
>> > > > > > otherwise, transmission data maybe will be lost.
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Hongtao Gao
>> > > > >
>> > > > > Apache SkyWalking && Apache ShardingSphere
>> > > > > Twitter, @hanahmily
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> Hongtao Gao
>>
>> Apache SkyWalking && Apache ShardingSphere
>> Twitter, @hanahmily
>>
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Sure, will call you at tomorrow 10am UTC+8.
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 下午10:56写道：
> Hi Sheng
> I prefer to schedule a hangout(maybe tomorrow) to discuss the details of my
> proposal, which seems not suitable to dig into it in this thread. This
> proposal inspires me to find out a new approach to propagate sw tracing
> context.
>
> Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
>
> > Hi Gao
> >
> > I am still confusing. If b3 used, then trace is the trace, there is
> nothing
> > related to what we do.
> > If application is instrumented by Zipkin SDK, then it is Zipkin's
> > transmission capability, not us.
> >
> > In the proposal, it is clear that our agent is going to propagate things
> > other injected and propagate them without touch. I can't see the link
> > between this and your mesh case.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> >
> > > Let me clarify what I'm talking about.
> > >
> > > In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
> > > propagation, which means they can extract trace context from downstream
> > and
> > > inject it to upstream.
> > >
> > > I want downside envoy sidecar instrumented by SkyWalking can wrap sw
> > trace
> > > context into B3 context to bypass target app/service to upside sidecar.
> > >
> > > When it happens, skywalking can trace application instrumented only by
> > > Zipkin in service mesh scenario.
> > >
> > > I prefer to Zipkin that because it seems the actual standard tracing
> > > protocol in the SM world. And we leverage it as a lower-level protocol
> to
> > > figure out tracing context overlay.
> > >
> > >
> > >
> > >
> > > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
> > wrote:
> > >
> > > > Hi Gao
> > > >
> > > > Look through your reply, are you talking about propagating zipkin
> > context
> > > > or skywalking context? Your case is not clear to me.
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > > >
> > > > > Look good to me. I think this feature is also fantastic to the
> > Service
> > > > Mesh
> > > > > scenario.
> > > > > For now, OAP ingests send and receive messages from the sidecar and
> > > uses
> > > > > statistic analysis to link them to a topology. There's no trace
> here.
> > > > >
> > > > > If we implement the transmission protocol, the service workload can
> > > > bypass
> > > > > skywalking protocol, sidecar creates a segment and sinks it to OAP
> > > > instead.
> > > > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> > > > > connected by the application who implement Zipkin protocol. I know
> it
> > > may
> > > > > be
> > > > > a big challenge for us, but from my perspective, it's not
> impossible
> > > > > totally.
> > > > >
> > > > > Thanks, Hongtao
> > > > >
> > > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > > > >
> > > > > > Hi dev team,
> > > > > >
> > > > > > I want to provide a message transmission future, which can be
> used
> > > for
> > > > > > message transmission in projects that already use skywalking.
> > > > > > This future of data transmission between multiple instances in
> the
> > > same
> > > > > > trace can be achieved by adding a new API of information
> > transmission
> > > > in
> > > > > > each language level, which does not need a dependency on the
> > existing
> > > > > > interface protocol.
> > > > > > For example, the execution process of a trace is A -> B -> C, Set
> > > > > transfer
> > > > > > content in A service, B and C services can obtain data through
> API.
> > > > > > This future is to use the message passing future of Skywaling,
> > > addition
> > > > > an
> > > > > > extension function. I'm not quite sure if it's appropriate to add
> > it
> > > to
> > > > > > Skywalking?
> > > > > >
> > > > > > Currently, it can be applied to these scenarios:
> > > > > > 1. For the existing old system, It's not convenient to revise the
> > > data
> > > > of
> > > > > > the existing protocol. Through embedding transparent data,
> > downstream
> > > > > > services can get data information.
> > > > > > 2. In the entry layer, such as zuul or other gateway services,
> > > > > > transparently transmits user identity or other customized
> > information
> > > > to
> > > > > > the downstream services.
> > > > > >
> > > > > > Implementation plan:
> > > > > > 1. Store transmission data as Map in the current segment.
> > > > > > 2. Create a new protocol header for data transmission (JSON)
> > > > > > 3. When the downstream system receives the new protocol, it will
> > > > > transform
> > > > > > the data back to Map and store to the segment
> > > > > > 4. I think this implementation should apply to trace and ignored
> > > > segment,
> > > > > > otherwise, transmission data maybe will be lost.
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Hongtao Gao
> > > > >
> > > > > Apache SkyWalking && Apache ShardingSphere
> > > > > Twitter, @hanahmily
> > > > >
> > > >
> > >
> >
>
>
> --
> Hongtao Gao
>
> Apache SkyWalking && Apache ShardingSphere
> Twitter, @hanahmily
>
Re: A new proposal of message transmission
Posted by Hongtao Gao <ha...@gmail.com>.
Hi Sheng
I prefer to schedule a hangout(maybe tomorrow) to discuss the details of my
proposal, which seems not suitable to dig into it in this thread. This
proposal inspires me to find out a new approach to propagate sw tracing
context.
Sheng Wu <wu...@gmail.com> 于2020年3月5日周四 上午10:35写道：
> Hi Gao
>
> I am still confusing. If b3 used, then trace is the trace, there is nothing
> related to what we do.
> If application is instrumented by Zipkin SDK, then it is Zipkin's
> transmission capability, not us.
>
> In the proposal, it is clear that our agent is going to propagate things
> other injected and propagate them without touch. I can't see the link
> between this and your mesh case.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
>
> > Let me clarify what I'm talking about.
> >
> > In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
> > propagation, which means they can extract trace context from downstream
> and
> > inject it to upstream.
> >
> > I want downside envoy sidecar instrumented by SkyWalking can wrap sw
> trace
> > context into B3 context to bypass target app/service to upside sidecar.
> >
> > When it happens, skywalking can trace application instrumented only by
> > Zipkin in service mesh scenario.
> >
> > I prefer to Zipkin that because it seems the actual standard tracing
> > protocol in the SM world. And we leverage it as a lower-level protocol to
> > figure out tracing context overlay.
> >
> >
> >
> >
> > On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com>
> wrote:
> >
> > > Hi Gao
> > >
> > > Look through your reply, are you talking about propagating zipkin
> context
> > > or skywalking context? Your case is not clear to me.
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> > >
> > > > Look good to me. I think this feature is also fantastic to the
> Service
> > > Mesh
> > > > scenario.
> > > > For now, OAP ingests send and receive messages from the sidecar and
> > uses
> > > > statistic analysis to link them to a topology. There's no trace here.
> > > >
> > > > If we implement the transmission protocol, the service workload can
> > > bypass
> > > > skywalking protocol, sidecar creates a segment and sinks it to OAP
> > > instead.
> > > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> > > > connected by the application who implement Zipkin protocol. I know it
> > may
> > > > be
> > > > a big challenge for us, but from my perspective, it's not impossible
> > > > totally.
> > > >
> > > > Thanks, Hongtao
> > > >
> > > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > > >
> > > > > Hi dev team,
> > > > >
> > > > > I want to provide a message transmission future, which can be used
> > for
> > > > > message transmission in projects that already use skywalking.
> > > > > This future of data transmission between multiple instances in the
> > same
> > > > > trace can be achieved by adding a new API of information
> transmission
> > > in
> > > > > each language level, which does not need a dependency on the
> existing
> > > > > interface protocol.
> > > > > For example, the execution process of a trace is A -> B -> C, Set
> > > > transfer
> > > > > content in A service, B and C services can obtain data through API.
> > > > > This future is to use the message passing future of Skywaling,
> > addition
> > > > an
> > > > > extension function. I'm not quite sure if it's appropriate to add
> it
> > to
> > > > > Skywalking?
> > > > >
> > > > > Currently, it can be applied to these scenarios:
> > > > > 1. For the existing old system, It's not convenient to revise the
> > data
> > > of
> > > > > the existing protocol. Through embedding transparent data,
> downstream
> > > > > services can get data information.
> > > > > 2. In the entry layer, such as zuul or other gateway services,
> > > > > transparently transmits user identity or other customized
> information
> > > to
> > > > > the downstream services.
> > > > >
> > > > > Implementation plan:
> > > > > 1. Store transmission data as Map in the current segment.
> > > > > 2. Create a new protocol header for data transmission (JSON)
> > > > > 3. When the downstream system receives the new protocol, it will
> > > > transform
> > > > > the data back to Map and store to the segment
> > > > > 4. I think this implementation should apply to trace and ignored
> > > segment,
> > > > > otherwise, transmission data maybe will be lost.
> > > > >
> > > >
> > > >
> > > > --
> > > > Hongtao Gao
> > > >
> > > > Apache SkyWalking && Apache ShardingSphere
> > > > Twitter, @hanahmily
> > > >
> > >
> >
>
--
Hongtao Gao
Apache SkyWalking && Apache ShardingSphere
Twitter, @hanahmily
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Hi Gao
I am still confusing. If b3 used, then trace is the trace, there is nothing
related to what we do.
If application is instrumented by Zipkin SDK, then it is Zipkin's
transmission capability, not us.
In the proposal, it is clear that our agent is going to propagate things
other injected and propagate them without touch. I can't see the link
between this and your mesh case.
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@gmail.com> 于2020年3月5日周四 上午10:29写道：
> Let me clarify what I'm talking about.
>
> In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
> propagation, which means they can extract trace context from downstream and
> inject it to upstream.
>
> I want downside envoy sidecar instrumented by SkyWalking can wrap sw trace
> context into B3 context to bypass target app/service to upside sidecar.
>
> When it happens, skywalking can trace application instrumented only by
> Zipkin in service mesh scenario.
>
> I prefer to Zipkin that because it seems the actual standard tracing
> protocol in the SM world. And we leverage it as a lower-level protocol to
> figure out tracing context overlay.
>
>
>
>
> On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com> wrote:
>
> > Hi Gao
> >
> > Look through your reply, are you talking about propagating zipkin context
> > or skywalking context? Your case is not clear to me.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> >
> > > Look good to me. I think this feature is also fantastic to the Service
> > Mesh
> > > scenario.
> > > For now, OAP ingests send and receive messages from the sidecar and
> uses
> > > statistic analysis to link them to a topology. There's no trace here.
> > >
> > > If we implement the transmission protocol, the service workload can
> > bypass
> > > skywalking protocol, sidecar creates a segment and sinks it to OAP
> > instead.
> > > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> > > connected by the application who implement Zipkin protocol. I know it
> may
> > > be
> > > a big challenge for us, but from my perspective, it's not impossible
> > > totally.
> > >
> > > Thanks, Hongtao
> > >
> > > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> > >
> > > > Hi dev team,
> > > >
> > > > I want to provide a message transmission future, which can be used
> for
> > > > message transmission in projects that already use skywalking.
> > > > This future of data transmission between multiple instances in the
> same
> > > > trace can be achieved by adding a new API of information transmission
> > in
> > > > each language level, which does not need a dependency on the existing
> > > > interface protocol.
> > > > For example, the execution process of a trace is A -> B -> C, Set
> > > transfer
> > > > content in A service, B and C services can obtain data through API.
> > > > This future is to use the message passing future of Skywaling,
> addition
> > > an
> > > > extension function. I'm not quite sure if it's appropriate to add it
> to
> > > > Skywalking?
> > > >
> > > > Currently, it can be applied to these scenarios:
> > > > 1. For the existing old system, It's not convenient to revise the
> data
> > of
> > > > the existing protocol. Through embedding transparent data, downstream
> > > > services can get data information.
> > > > 2. In the entry layer, such as zuul or other gateway services,
> > > > transparently transmits user identity or other customized information
> > to
> > > > the downstream services.
> > > >
> > > > Implementation plan:
> > > > 1. Store transmission data as Map in the current segment.
> > > > 2. Create a new protocol header for data transmission (JSON)
> > > > 3. When the downstream system receives the new protocol, it will
> > > transform
> > > > the data back to Map and store to the segment
> > > > 4. I think this implementation should apply to trace and ignored
> > segment,
> > > > otherwise, transmission data maybe will be lost.
> > > >
> > >
> > >
> > > --
> > > Hongtao Gao
> > >
> > > Apache SkyWalking && Apache ShardingSphere
> > > Twitter, @hanahmily
> > >
> >
>
Re: A new proposal of message transmission
Posted by Hongtao Gao <ha...@gmail.com>.
Let me clarify what I'm talking about.
In the Istio demo, just like Bookinfo, all of the apps implement Zipkin
propagation, which means they can extract trace context from downstream and
inject it to upstream.
I want downside envoy sidecar instrumented by SkyWalking can wrap sw trace
context into B3 context to bypass target app/service to upside sidecar.
When it happens, skywalking can trace application instrumented only by
Zipkin in service mesh scenario.
I prefer to Zipkin that because it seems the actual standard tracing
protocol in the SM world. And we leverage it as a lower-level protocol to
figure out tracing context overlay.
On Wed, Mar 4, 2020, 10:41 PM Sheng Wu <wu...@gmail.com> wrote:
> Hi Gao
>
> Look through your reply, are you talking about propagating zipkin context
> or skywalking context? Your case is not clear to me.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
>
> > Look good to me. I think this feature is also fantastic to the Service
> Mesh
> > scenario.
> > For now, OAP ingests send and receive messages from the sidecar and uses
> > statistic analysis to link them to a topology. There's no trace here.
> >
> > If we implement the transmission protocol, the service workload can
> bypass
> > skywalking protocol, sidecar creates a segment and sinks it to OAP
> instead.
> > And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> > connected by the application who implement Zipkin protocol. I know it may
> > be
> > a big challenge for us, but from my perspective, it's not impossible
> > totally.
> >
> > Thanks, Hongtao
> >
> > han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> >
> > > Hi dev team,
> > >
> > > I want to provide a message transmission future, which can be used for
> > > message transmission in projects that already use skywalking.
> > > This future of data transmission between multiple instances in the same
> > > trace can be achieved by adding a new API of information transmission
> in
> > > each language level, which does not need a dependency on the existing
> > > interface protocol.
> > > For example, the execution process of a trace is A -> B -> C, Set
> > transfer
> > > content in A service, B and C services can obtain data through API.
> > > This future is to use the message passing future of Skywaling, addition
> > an
> > > extension function. I'm not quite sure if it's appropriate to add it to
> > > Skywalking?
> > >
> > > Currently, it can be applied to these scenarios:
> > > 1. For the existing old system, It's not convenient to revise the data
> of
> > > the existing protocol. Through embedding transparent data, downstream
> > > services can get data information.
> > > 2. In the entry layer, such as zuul or other gateway services,
> > > transparently transmits user identity or other customized information
> to
> > > the downstream services.
> > >
> > > Implementation plan:
> > > 1. Store transmission data as Map in the current segment.
> > > 2. Create a new protocol header for data transmission (JSON)
> > > 3. When the downstream system receives the new protocol, it will
> > transform
> > > the data back to Map and store to the segment
> > > 4. I think this implementation should apply to trace and ignored
> segment,
> > > otherwise, transmission data maybe will be lost.
> > >
> >
> >
> > --
> > Hongtao Gao
> >
> > Apache SkyWalking && Apache ShardingSphere
> > Twitter, @hanahmily
> >
>
Re: A new proposal of message transmission
Posted by Sheng Wu <wu...@gmail.com>.
Hi Gao
Look through your reply, are you talking about propagating zipkin context
or skywalking context? Your case is not clear to me.
Sheng Wu 吴晟
Twitter, wusheng1108
Hongtao Gao <ha...@gmail.com> 于2020年3月4日周三 下午10:31写道：
> Look good to me. I think this feature is also fantastic to the Service Mesh
> scenario.
> For now, OAP ingests send and receive messages from the sidecar and uses
> statistic analysis to link them to a topology. There's no trace here.
>
> If we implement the transmission protocol, the service workload can bypass
> skywalking protocol, sidecar creates a segment and sinks it to OAP instead.
> And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
> connected by the application who implement Zipkin protocol. I know it may
> be
> a big challenge for us, but from my perspective, it's not impossible
> totally.
>
> Thanks, Hongtao
>
> han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
>
> > Hi dev team,
> >
> > I want to provide a message transmission future, which can be used for
> > message transmission in projects that already use skywalking.
> > This future of data transmission between multiple instances in the same
> > trace can be achieved by adding a new API of information transmission in
> > each language level, which does not need a dependency on the existing
> > interface protocol.
> > For example, the execution process of a trace is A -> B -> C, Set
> transfer
> > content in A service, B and C services can obtain data through API.
> > This future is to use the message passing future of Skywaling, addition
> an
> > extension function. I'm not quite sure if it's appropriate to add it to
> > Skywalking?
> >
> > Currently, it can be applied to these scenarios:
> > 1. For the existing old system, It's not convenient to revise the data of
> > the existing protocol. Through embedding transparent data, downstream
> > services can get data information.
> > 2. In the entry layer, such as zuul or other gateway services,
> > transparently transmits user identity or other customized information to
> > the downstream services.
> >
> > Implementation plan:
> > 1. Store transmission data as Map in the current segment.
> > 2. Create a new protocol header for data transmission (JSON)
> > 3. When the downstream system receives the new protocol, it will
> transform
> > the data back to Map and store to the segment
> > 4. I think this implementation should apply to trace and ignored segment,
> > otherwise, transmission data maybe will be lost.
> >
>
>
> --
> Hongtao Gao
>
> Apache SkyWalking && Apache ShardingSphere
> Twitter, @hanahmily
>
Re: A new proposal of message transmission
Posted by Hongtao Gao <ha...@gmail.com>.
Look good to me. I think this feature is also fantastic to the Service Mesh
scenario.
For now, OAP ingests send and receive messages from the sidecar and uses
statistic analysis to link them to a topology. There's no trace here.
If we implement the transmission protocol, the service workload can bypass
skywalking protocol, sidecar creates a segment and sinks it to OAP instead.
And furthermore, if we figure out sw6 over B3(Zipkin), our trace is
connected by the application who implement Zipkin protocol. I know it may
be
a big challenge for us, but from my perspective, it's not impossible
totally.
Thanks, Hongtao
han liu <mr...@gmail.com> 于2020年3月4日周三 下午5:06写道：
> Hi dev team,
>
> I want to provide a message transmission future, which can be used for
> message transmission in projects that already use skywalking.
> This future of data transmission between multiple instances in the same
> trace can be achieved by adding a new API of information transmission in
> each language level, which does not need a dependency on the existing
> interface protocol.
> For example, the execution process of a trace is A -> B -> C, Set transfer
> content in A service, B and C services can obtain data through API.
> This future is to use the message passing future of Skywaling, addition an
> extension function. I'm not quite sure if it's appropriate to add it to
> Skywalking?
>
> Currently, it can be applied to these scenarios:
> 1. For the existing old system, It's not convenient to revise the data of
> the existing protocol. Through embedding transparent data, downstream
> services can get data information.
> 2. In the entry layer, such as zuul or other gateway services,
> transparently transmits user identity or other customized information to
> the downstream services.
>
> Implementation plan:
> 1. Store transmission data as Map in the current segment.
> 2. Create a new protocol header for data transmission (JSON)
> 3. When the downstream system receives the new protocol, it will transform
> the data back to Map and store to the segment
> 4. I think this implementation should apply to trace and ignored segment,
> otherwise, transmission data maybe will be lost.
>
--
Hongtao Gao
Apache SkyWalking && Apache ShardingSphere
Twitter, @hanahmily