You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Tsuyoshi Ozawa <oz...@apache.org> on 2018/08/21 10:43:54 UTC
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Thanks for starting discussion, Stack.
The ZipKin seems to be coming to the Apache Incubator. As Andrew
Purtell said on HADOOP-15566, it would be good option since there is
no problem about licenses.
https://wiki.apache.org/incubator/ZipkinProposal
Stack, do you have any knowledge about differences between Zipkin and
HTrace? Might measurable performance overhead be observed still in
Zipkin?
To decrease the overhead, we need to do additional work like ftrace,
well known dtrace implementation in Linux kernel. If I understand
correctly, ftrace replace its function calls with NOP operations of
CPU instruction when it is disabled. This ensures the lower overhead
by the tracer. By replacing the function calls for tracing to JVM's
NOP operation, can we achieve the minimum overhead?
Regards
- Tsuyoshi
On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com> wrote:
>
> Most of code coverage tools can instrument java classes without make any
> source code changes, but tracing distributed system is more involved because
> code execution via network interactions are not easy to match up.
> All interactions between sender and receiver have some form of session id
> or sequence id. Hadoop had some logic to assist the stitching of distributed
> interactions together in clienttrace log. This information seems to have been
> lost in the last 5-6 years of Hadoop evolutions. Htrace is invented to fill the void
> left behind by clienttrace as a programmable API to send out useful tracing data for
> downstream analytical program to visualize the interaction.
>
> Large companies have common practice to enforce logging the session id, and
> write homebrew tools to stitch together debugging logic for a specific software.
> There are also growing set of tools from Splunk or similar companies to write
> analytical tools to stitch the views together. Hadoop does not seem to be on
> top of the list for those company to implement the tracing because Hadoop
> networking layer is complex and changed more frequently than desired.
>
> If we go back to logging approach, instead of API approach, it will help
> someone to write the analytical program someday. The danger of logging
> approach is that It is boring to write LOG.debug() everywhere, and we
> often forgot about it, and log entries are removed.
>
> API approach can work, if real time interactive tracing can be done.
> However, this is hard to realize in Hadoop because massive amount of
> parallel data is difficult to aggregate at real time without hitting timeout.
> It has a higher chance to require changes to network protocol that might cause
> more headache than it's worth. I am in favor of removing Htrace support
> and redo distributed tracing using logging approach.
>
> Regards,
> Eric
>
> On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
>
> There is a healthy discussion going on over in HADOOP-15566 on tracing
> in the Hadoop ecosystem. It would sit better on a mailing list than in
> comments up on JIRA so here's an attempt at porting the chat here.
>
> Background/Context: Bits of Hadoop and HBase had Apache HTrace trace
> points added. HTrace was formerly "incubating" at Apache but has since
> been retired, moved to Apache Attic. HTrace and the efforts at
> instrumenting Hadoop wilted for want of attention/resourcing. Our Todd
> Lipcon noticed that the HTrace instrumentation can add friction on
> some code paths so can actually be harmful even when disabled. The
> natural follow-on is that we should rip out tracings of a "dead"
> project. This then beggars the question, should something replace it
> and if so what? This is where HADOOP-15566 is at currently.
>
> HTrace took two or three runs, led by various Heros, at building a
> trace lib for Hadoop (first). It was trying to build the trace lib, a
> store, and a visualizer. Always, it had a mechanism for dumping the
> traces out to external systems for storage and viewing (e.g. Zipkin).
> HTrace started when there was little else but the, you guessed it,
> Google paper that described the Dapper system they had internally.
> Since then, the world of tracing has come on in leaps and bounds with
> healthy alternatives, communities, and even commercialization.
>
> If interested, take a read over HADOOP-15566. Will try and encourage
> participants to move the chat here.
>
> Thanks,
> St.Ack
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <an...@gmail.com>.
And, sorry if not clear, only source level compatibility is needed. The goal wouldn't be drop in binary compatibility for user convenience, it would be source compatibility so we don't have to replace all of the HTrace instrumentation points cross stack in the short term, so, stack developer and release management convenience.
That aside, I have no idea if zipkin would want to ship a HTrace facade.
> On Aug 21, 2018, at 11:42 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> I was assuming taking the HTrace API implementation, removing all code from the methods, and reimplementing with Brave wouldn't face insurmountable challenges, especially given the result is only meant for near term use, but I can't say I've tried nor looked into it in details. Was thinking that would be the first step of an effort in this regard. If not possible, we will have to do this major lift where everyone reimplements tracing cross stack right away, rather than pivot on a transitional facade.
>
>>> On Aug 21, 2018, at 11:18 AM, Stack <st...@duboce.net> wrote:
>>>
>>> On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
>>>
>>> What if someone built a HTrace facade for Zipkin / Brave?
>>
>>
>> I like the idea but taking a look, HTrace does static dispatch. I was
>> thinking that precludes our being able to do a facade. I would love to hear
>> otherwise.
>> Thanks,
>> S
>>
>>
>>> Hadoop, HBase,
>>> Phoenix, and other HTrace API users would still need to move away from
>>> embedding HTrace instrumentation points to whatever is the normal API of
>>> the accepted replacement, but such a facade would give you a drop in
>>> replacement requiring no code changes to currently shipping code lines, and
>>> some time to do a hopefully coordinated replacement involving all upstreams
>>> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
>>> that option and the impending incubation here at the ASF will make it quite
>>> attractive, I think.
>>>
>>>
>>>>> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>>>>>
>>>>> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>> Thanks for starting discussion, Stack.
>>>>>
>>>>> The ZipKin seems to be coming to the Apache Incubator. As Andrew
>>>>> Purtell said on HADOOP-15566, it would be good option since there is
>>>>> no problem about licenses.
>>>>> https://wiki.apache.org/incubator/ZipkinProposal
>>>>>
>>>>>
>>>> Yes. This is nice to see.
>>>>
>>>>
>>>>
>>>>> Stack, do you have any knowledge about differences between Zipkin and
>>>>> HTrace? Might measurable performance overhead be observed still in
>>>>> Zipkin?
>>>>>
>>>>>
>>>> I've not measured to see if disabled trace points are friction-free.
>>>> Perhaps someone else has?
>>>>
>>>>
>>>>
>>>>> To decrease the overhead, we need to do additional work like ftrace,
>>>>> well known dtrace implementation in Linux kernel. If I understand
>>>>> correctly, ftrace replace its function calls with NOP operations of
>>>>> CPU instruction when it is disabled. This ensures the lower overhead
>>>>> by the tracer. By replacing the function calls for tracing to JVM's
>>>>> NOP operation, can we achieve the minimum overhead?
>>>>>
>>>>>
>>>> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
>>>> context, we should be able to get away with something less exotic.
>>>>
>>>> Thanks Tsuyoshi,
>>>> S
>>>>
>>>>
>>>>
>>>>
>>>>> Regards
>>>>> - Tsuyoshi
>>>>> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
>>> wrote:
>>>>>>
>>>>>> Most of code coverage tools can instrument java classes without make
>>>> any
>>>>>> source code changes, but tracing distributed system is more involved
>>>>> because
>>>>>> code execution via network interactions are not easy to match up.
>>>>>> All interactions between sender and receiver have some form of
>>> session
>>>> id
>>>>>> or sequence id. Hadoop had some logic to assist the stitching of
>>>>> distributed
>>>>>> interactions together in clienttrace log. This information seems to
>>>>> have been
>>>>>> lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
>>> to
>>>>> fill the void
>>>>>> left behind by clienttrace as a programmable API to send out useful
>>>>> tracing data for
>>>>>> downstream analytical program to visualize the interaction.
>>>>>>
>>>>>> Large companies have common practice to enforce logging the session
>>> id,
>>>>> and
>>>>>> write homebrew tools to stitch together debugging logic for a
>>> specific
>>>>> software.
>>>>>> There are also growing set of tools from Splunk or similar companies
>>> to
>>>>> write
>>>>>> analytical tools to stitch the views together. Hadoop does not seem
>>> to
>>>>> be on
>>>>>> top of the list for those company to implement the tracing because
>>>> Hadoop
>>>>>> networking layer is complex and changed more frequently than desired.
>>>>>>
>>>>>> If we go back to logging approach, instead of API approach, it will
>>>> help
>>>>>> someone to write the analytical program someday. The danger of
>>> logging
>>>>>> approach is that It is boring to write LOG.debug() everywhere, and we
>>>>>> often forgot about it, and log entries are removed.
>>>>>>
>>>>>> API approach can work, if real time interactive tracing can be done.
>>>>>> However, this is hard to realize in Hadoop because massive amount of
>>>>>> parallel data is difficult to aggregate at real time without hitting
>>>>> timeout.
>>>>>> It has a higher chance to require changes to network protocol that
>>>> might
>>>>> cause
>>>>>> more headache than it's worth. I am in favor of removing Htrace
>>>> support
>>>>>> and redo distributed tracing using logging approach.
>>>>>>
>>>>>> Regards,
>>>>>> Eric
>>>>>>
>>>>>> On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
>>>>>>
>>>>>> There is a healthy discussion going on over in HADOOP-15566 on
>>>>> tracing
>>>>>> in the Hadoop ecosystem. It would sit better on a mailing list
>>> than
>>>>> in
>>>>>> comments up on JIRA so here's an attempt at porting the chat
>>> here.
>>>>>>
>>>>>> Background/Context: Bits of Hadoop and HBase had Apache HTrace
>>>> trace
>>>>>> points added. HTrace was formerly "incubating" at Apache but has
>>>>> since
>>>>>> been retired, moved to Apache Attic. HTrace and the efforts at
>>>>>> instrumenting Hadoop wilted for want of attention/resourcing. Our
>>>>> Todd
>>>>>> Lipcon noticed that the HTrace instrumentation can add friction
>>> on
>>>>>> some code paths so can actually be harmful even when disabled.
>>> The
>>>>>> natural follow-on is that we should rip out tracings of a "dead"
>>>>>> project. This then beggars the question, should something replace
>>>> it
>>>>>> and if so what? This is where HADOOP-15566 is at currently.
>>>>>>
>>>>>> HTrace took two or three runs, led by various Heros, at building
>>> a
>>>>>> trace lib for Hadoop (first). It was trying to build the trace
>>>> lib, a
>>>>>> store, and a visualizer. Always, it had a mechanism for dumping
>>> the
>>>>>> traces out to external systems for storage and viewing (e.g.
>>>> Zipkin).
>>>>>> HTrace started when there was little else but the, you guessed
>>> it,
>>>>>> Google paper that described the Dapper system they had
>>> internally.
>>>>>> Since then, the world of tracing has come on in leaps and bounds
>>>> with
>>>>>> healthy alternatives, communities, and even commercialization.
>>>>>>
>>>>>> If interested, take a read over HADOOP-15566. Will try and
>>>> encourage
>>>>>> participants to move the chat here.
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail:
>>> common-dev-help@hadoop.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrew
>>>
>>> Words like orphans lost among the crosstalk, meaning torn from truth's
>>> decrepit hands
>>> - A23, Crosstalk
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <an...@gmail.com>.
And, sorry if not clear, only source level compatibility is needed. The goal wouldn't be drop in binary compatibility for user convenience, it would be source compatibility so we don't have to replace all of the HTrace instrumentation points cross stack in the short term, so, stack developer and release management convenience.
That aside, I have no idea if zipkin would want to ship a HTrace facade.
> On Aug 21, 2018, at 11:42 AM, Andrew Purtell <an...@gmail.com> wrote:
>
> I was assuming taking the HTrace API implementation, removing all code from the methods, and reimplementing with Brave wouldn't face insurmountable challenges, especially given the result is only meant for near term use, but I can't say I've tried nor looked into it in details. Was thinking that would be the first step of an effort in this regard. If not possible, we will have to do this major lift where everyone reimplements tracing cross stack right away, rather than pivot on a transitional facade.
>
>>> On Aug 21, 2018, at 11:18 AM, Stack <st...@duboce.net> wrote:
>>>
>>> On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
>>>
>>> What if someone built a HTrace facade for Zipkin / Brave?
>>
>>
>> I like the idea but taking a look, HTrace does static dispatch. I was
>> thinking that precludes our being able to do a facade. I would love to hear
>> otherwise.
>> Thanks,
>> S
>>
>>
>>> Hadoop, HBase,
>>> Phoenix, and other HTrace API users would still need to move away from
>>> embedding HTrace instrumentation points to whatever is the normal API of
>>> the accepted replacement, but such a facade would give you a drop in
>>> replacement requiring no code changes to currently shipping code lines, and
>>> some time to do a hopefully coordinated replacement involving all upstreams
>>> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
>>> that option and the impending incubation here at the ASF will make it quite
>>> attractive, I think.
>>>
>>>
>>>>> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>>>>>
>>>>> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>> Thanks for starting discussion, Stack.
>>>>>
>>>>> The ZipKin seems to be coming to the Apache Incubator. As Andrew
>>>>> Purtell said on HADOOP-15566, it would be good option since there is
>>>>> no problem about licenses.
>>>>> https://wiki.apache.org/incubator/ZipkinProposal
>>>>>
>>>>>
>>>> Yes. This is nice to see.
>>>>
>>>>
>>>>
>>>>> Stack, do you have any knowledge about differences between Zipkin and
>>>>> HTrace? Might measurable performance overhead be observed still in
>>>>> Zipkin?
>>>>>
>>>>>
>>>> I've not measured to see if disabled trace points are friction-free.
>>>> Perhaps someone else has?
>>>>
>>>>
>>>>
>>>>> To decrease the overhead, we need to do additional work like ftrace,
>>>>> well known dtrace implementation in Linux kernel. If I understand
>>>>> correctly, ftrace replace its function calls with NOP operations of
>>>>> CPU instruction when it is disabled. This ensures the lower overhead
>>>>> by the tracer. By replacing the function calls for tracing to JVM's
>>>>> NOP operation, can we achieve the minimum overhead?
>>>>>
>>>>>
>>>> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
>>>> context, we should be able to get away with something less exotic.
>>>>
>>>> Thanks Tsuyoshi,
>>>> S
>>>>
>>>>
>>>>
>>>>
>>>>> Regards
>>>>> - Tsuyoshi
>>>>> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
>>> wrote:
>>>>>>
>>>>>> Most of code coverage tools can instrument java classes without make
>>>> any
>>>>>> source code changes, but tracing distributed system is more involved
>>>>> because
>>>>>> code execution via network interactions are not easy to match up.
>>>>>> All interactions between sender and receiver have some form of
>>> session
>>>> id
>>>>>> or sequence id. Hadoop had some logic to assist the stitching of
>>>>> distributed
>>>>>> interactions together in clienttrace log. This information seems to
>>>>> have been
>>>>>> lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
>>> to
>>>>> fill the void
>>>>>> left behind by clienttrace as a programmable API to send out useful
>>>>> tracing data for
>>>>>> downstream analytical program to visualize the interaction.
>>>>>>
>>>>>> Large companies have common practice to enforce logging the session
>>> id,
>>>>> and
>>>>>> write homebrew tools to stitch together debugging logic for a
>>> specific
>>>>> software.
>>>>>> There are also growing set of tools from Splunk or similar companies
>>> to
>>>>> write
>>>>>> analytical tools to stitch the views together. Hadoop does not seem
>>> to
>>>>> be on
>>>>>> top of the list for those company to implement the tracing because
>>>> Hadoop
>>>>>> networking layer is complex and changed more frequently than desired.
>>>>>>
>>>>>> If we go back to logging approach, instead of API approach, it will
>>>> help
>>>>>> someone to write the analytical program someday. The danger of
>>> logging
>>>>>> approach is that It is boring to write LOG.debug() everywhere, and we
>>>>>> often forgot about it, and log entries are removed.
>>>>>>
>>>>>> API approach can work, if real time interactive tracing can be done.
>>>>>> However, this is hard to realize in Hadoop because massive amount of
>>>>>> parallel data is difficult to aggregate at real time without hitting
>>>>> timeout.
>>>>>> It has a higher chance to require changes to network protocol that
>>>> might
>>>>> cause
>>>>>> more headache than it's worth. I am in favor of removing Htrace
>>>> support
>>>>>> and redo distributed tracing using logging approach.
>>>>>>
>>>>>> Regards,
>>>>>> Eric
>>>>>>
>>>>>> On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
>>>>>>
>>>>>> There is a healthy discussion going on over in HADOOP-15566 on
>>>>> tracing
>>>>>> in the Hadoop ecosystem. It would sit better on a mailing list
>>> than
>>>>> in
>>>>>> comments up on JIRA so here's an attempt at porting the chat
>>> here.
>>>>>>
>>>>>> Background/Context: Bits of Hadoop and HBase had Apache HTrace
>>>> trace
>>>>>> points added. HTrace was formerly "incubating" at Apache but has
>>>>> since
>>>>>> been retired, moved to Apache Attic. HTrace and the efforts at
>>>>>> instrumenting Hadoop wilted for want of attention/resourcing. Our
>>>>> Todd
>>>>>> Lipcon noticed that the HTrace instrumentation can add friction
>>> on
>>>>>> some code paths so can actually be harmful even when disabled.
>>> The
>>>>>> natural follow-on is that we should rip out tracings of a "dead"
>>>>>> project. This then beggars the question, should something replace
>>>> it
>>>>>> and if so what? This is where HADOOP-15566 is at currently.
>>>>>>
>>>>>> HTrace took two or three runs, led by various Heros, at building
>>> a
>>>>>> trace lib for Hadoop (first). It was trying to build the trace
>>>> lib, a
>>>>>> store, and a visualizer. Always, it had a mechanism for dumping
>>> the
>>>>>> traces out to external systems for storage and viewing (e.g.
>>>> Zipkin).
>>>>>> HTrace started when there was little else but the, you guessed
>>> it,
>>>>>> Google paper that described the Dapper system they had
>>> internally.
>>>>>> Since then, the world of tracing has come on in leaps and bounds
>>>> with
>>>>>> healthy alternatives, communities, and even commercialization.
>>>>>>
>>>>>> If interested, take a read over HADOOP-15566. Will try and
>>>> encourage
>>>>>> participants to move the chat here.
>>>>>>
>>>>>> Thanks,
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail:
>>> common-dev-help@hadoop.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Andrew
>>>
>>> Words like orphans lost among the crosstalk, meaning torn from truth's
>>> decrepit hands
>>> - A23, Crosstalk
>>>
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <an...@gmail.com>.
I was assuming taking the HTrace API implementation, removing all code from the methods, and reimplementing with Brave wouldn't face insurmountable challenges, especially given the result is only meant for near term use, but I can't say I've tried nor looked into it in details. Was thinking that would be the first step of an effort in this regard. If not possible, we will have to do this major lift where everyone reimplements tracing cross stack right away, rather than pivot on a transitional facade.
> On Aug 21, 2018, at 11:18 AM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
>>
>> What if someone built a HTrace facade for Zipkin / Brave?
>
>
> I like the idea but taking a look, HTrace does static dispatch. I was
> thinking that precludes our being able to do a facade. I would love to hear
> otherwise.
> Thanks,
> S
>
>
>> Hadoop, HBase,
>> Phoenix, and other HTrace API users would still need to move away from
>> embedding HTrace instrumentation points to whatever is the normal API of
>> the accepted replacement, but such a facade would give you a drop in
>> replacement requiring no code changes to currently shipping code lines, and
>> some time to do a hopefully coordinated replacement involving all upstreams
>> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
>> that option and the impending incubation here at the ASF will make it quite
>> attractive, I think.
>>
>>
>>> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>>>
>>>> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>> Thanks for starting discussion, Stack.
>>>>
>>>> The ZipKin seems to be coming to the Apache Incubator. As Andrew
>>>> Purtell said on HADOOP-15566, it would be good option since there is
>>>> no problem about licenses.
>>>> https://wiki.apache.org/incubator/ZipkinProposal
>>>>
>>>>
>>> Yes. This is nice to see.
>>>
>>>
>>>
>>>> Stack, do you have any knowledge about differences between Zipkin and
>>>> HTrace? Might measurable performance overhead be observed still in
>>>> Zipkin?
>>>>
>>>>
>>> I've not measured to see if disabled trace points are friction-free.
>>> Perhaps someone else has?
>>>
>>>
>>>
>>>> To decrease the overhead, we need to do additional work like ftrace,
>>>> well known dtrace implementation in Linux kernel. If I understand
>>>> correctly, ftrace replace its function calls with NOP operations of
>>>> CPU instruction when it is disabled. This ensures the lower overhead
>>>> by the tracer. By replacing the function calls for tracing to JVM's
>>>> NOP operation, can we achieve the minimum overhead?
>>>>
>>>>
>>> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
>>> context, we should be able to get away with something less exotic.
>>>
>>> Thanks Tsuyoshi,
>>> S
>>>
>>>
>>>
>>>
>>>> Regards
>>>> - Tsuyoshi
>>>> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
>> wrote:
>>>>>
>>>>> Most of code coverage tools can instrument java classes without make
>>> any
>>>>> source code changes, but tracing distributed system is more involved
>>>> because
>>>>> code execution via network interactions are not easy to match up.
>>>>> All interactions between sender and receiver have some form of
>> session
>>> id
>>>>> or sequence id. Hadoop had some logic to assist the stitching of
>>>> distributed
>>>>> interactions together in clienttrace log. This information seems to
>>>> have been
>>>>> lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
>> to
>>>> fill the void
>>>>> left behind by clienttrace as a programmable API to send out useful
>>>> tracing data for
>>>>> downstream analytical program to visualize the interaction.
>>>>>
>>>>> Large companies have common practice to enforce logging the session
>> id,
>>>> and
>>>>> write homebrew tools to stitch together debugging logic for a
>> specific
>>>> software.
>>>>> There are also growing set of tools from Splunk or similar companies
>> to
>>>> write
>>>>> analytical tools to stitch the views together. Hadoop does not seem
>> to
>>>> be on
>>>>> top of the list for those company to implement the tracing because
>>> Hadoop
>>>>> networking layer is complex and changed more frequently than desired.
>>>>>
>>>>> If we go back to logging approach, instead of API approach, it will
>>> help
>>>>> someone to write the analytical program someday. The danger of
>> logging
>>>>> approach is that It is boring to write LOG.debug() everywhere, and we
>>>>> often forgot about it, and log entries are removed.
>>>>>
>>>>> API approach can work, if real time interactive tracing can be done.
>>>>> However, this is hard to realize in Hadoop because massive amount of
>>>>> parallel data is difficult to aggregate at real time without hitting
>>>> timeout.
>>>>> It has a higher chance to require changes to network protocol that
>>> might
>>>> cause
>>>>> more headache than it's worth. I am in favor of removing Htrace
>>> support
>>>>> and redo distributed tracing using logging approach.
>>>>>
>>>>> Regards,
>>>>> Eric
>>>>>
>>>>> On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
>>>>>
>>>>> There is a healthy discussion going on over in HADOOP-15566 on
>>>> tracing
>>>>> in the Hadoop ecosystem. It would sit better on a mailing list
>> than
>>>> in
>>>>> comments up on JIRA so here's an attempt at porting the chat
>> here.
>>>>>
>>>>> Background/Context: Bits of Hadoop and HBase had Apache HTrace
>>> trace
>>>>> points added. HTrace was formerly "incubating" at Apache but has
>>>> since
>>>>> been retired, moved to Apache Attic. HTrace and the efforts at
>>>>> instrumenting Hadoop wilted for want of attention/resourcing. Our
>>>> Todd
>>>>> Lipcon noticed that the HTrace instrumentation can add friction
>> on
>>>>> some code paths so can actually be harmful even when disabled.
>> The
>>>>> natural follow-on is that we should rip out tracings of a "dead"
>>>>> project. This then beggars the question, should something replace
>>> it
>>>>> and if so what? This is where HADOOP-15566 is at currently.
>>>>>
>>>>> HTrace took two or three runs, led by various Heros, at building
>> a
>>>>> trace lib for Hadoop (first). It was trying to build the trace
>>> lib, a
>>>>> store, and a visualizer. Always, it had a mechanism for dumping
>> the
>>>>> traces out to external systems for storage and viewing (e.g.
>>> Zipkin).
>>>>> HTrace started when there was little else but the, you guessed
>> it,
>>>>> Google paper that described the Dapper system they had
>> internally.
>>>>> Since then, the world of tracing has come on in leaps and bounds
>>> with
>>>>> healthy alternatives, communities, and even commercialization.
>>>>>
>>>>> If interested, take a read over HADOOP-15566. Will try and
>>> encourage
>>>>> participants to move the chat here.
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <an...@gmail.com>.
I was assuming taking the HTrace API implementation, removing all code from the methods, and reimplementing with Brave wouldn't face insurmountable challenges, especially given the result is only meant for near term use, but I can't say I've tried nor looked into it in details. Was thinking that would be the first step of an effort in this regard. If not possible, we will have to do this major lift where everyone reimplements tracing cross stack right away, rather than pivot on a transitional facade.
> On Aug 21, 2018, at 11:18 AM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
>>
>> What if someone built a HTrace facade for Zipkin / Brave?
>
>
> I like the idea but taking a look, HTrace does static dispatch. I was
> thinking that precludes our being able to do a facade. I would love to hear
> otherwise.
> Thanks,
> S
>
>
>> Hadoop, HBase,
>> Phoenix, and other HTrace API users would still need to move away from
>> embedding HTrace instrumentation points to whatever is the normal API of
>> the accepted replacement, but such a facade would give you a drop in
>> replacement requiring no code changes to currently shipping code lines, and
>> some time to do a hopefully coordinated replacement involving all upstreams
>> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
>> that option and the impending incubation here at the ASF will make it quite
>> attractive, I think.
>>
>>
>>> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>>>
>>>> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>> Thanks for starting discussion, Stack.
>>>>
>>>> The ZipKin seems to be coming to the Apache Incubator. As Andrew
>>>> Purtell said on HADOOP-15566, it would be good option since there is
>>>> no problem about licenses.
>>>> https://wiki.apache.org/incubator/ZipkinProposal
>>>>
>>>>
>>> Yes. This is nice to see.
>>>
>>>
>>>
>>>> Stack, do you have any knowledge about differences between Zipkin and
>>>> HTrace? Might measurable performance overhead be observed still in
>>>> Zipkin?
>>>>
>>>>
>>> I've not measured to see if disabled trace points are friction-free.
>>> Perhaps someone else has?
>>>
>>>
>>>
>>>> To decrease the overhead, we need to do additional work like ftrace,
>>>> well known dtrace implementation in Linux kernel. If I understand
>>>> correctly, ftrace replace its function calls with NOP operations of
>>>> CPU instruction when it is disabled. This ensures the lower overhead
>>>> by the tracer. By replacing the function calls for tracing to JVM's
>>>> NOP operation, can we achieve the minimum overhead?
>>>>
>>>>
>>> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
>>> context, we should be able to get away with something less exotic.
>>>
>>> Thanks Tsuyoshi,
>>> S
>>>
>>>
>>>
>>>
>>>> Regards
>>>> - Tsuyoshi
>>>> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
>> wrote:
>>>>>
>>>>> Most of code coverage tools can instrument java classes without make
>>> any
>>>>> source code changes, but tracing distributed system is more involved
>>>> because
>>>>> code execution via network interactions are not easy to match up.
>>>>> All interactions between sender and receiver have some form of
>> session
>>> id
>>>>> or sequence id. Hadoop had some logic to assist the stitching of
>>>> distributed
>>>>> interactions together in clienttrace log. This information seems to
>>>> have been
>>>>> lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
>> to
>>>> fill the void
>>>>> left behind by clienttrace as a programmable API to send out useful
>>>> tracing data for
>>>>> downstream analytical program to visualize the interaction.
>>>>>
>>>>> Large companies have common practice to enforce logging the session
>> id,
>>>> and
>>>>> write homebrew tools to stitch together debugging logic for a
>> specific
>>>> software.
>>>>> There are also growing set of tools from Splunk or similar companies
>> to
>>>> write
>>>>> analytical tools to stitch the views together. Hadoop does not seem
>> to
>>>> be on
>>>>> top of the list for those company to implement the tracing because
>>> Hadoop
>>>>> networking layer is complex and changed more frequently than desired.
>>>>>
>>>>> If we go back to logging approach, instead of API approach, it will
>>> help
>>>>> someone to write the analytical program someday. The danger of
>> logging
>>>>> approach is that It is boring to write LOG.debug() everywhere, and we
>>>>> often forgot about it, and log entries are removed.
>>>>>
>>>>> API approach can work, if real time interactive tracing can be done.
>>>>> However, this is hard to realize in Hadoop because massive amount of
>>>>> parallel data is difficult to aggregate at real time without hitting
>>>> timeout.
>>>>> It has a higher chance to require changes to network protocol that
>>> might
>>>> cause
>>>>> more headache than it's worth. I am in favor of removing Htrace
>>> support
>>>>> and redo distributed tracing using logging approach.
>>>>>
>>>>> Regards,
>>>>> Eric
>>>>>
>>>>> On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
>>>>>
>>>>> There is a healthy discussion going on over in HADOOP-15566 on
>>>> tracing
>>>>> in the Hadoop ecosystem. It would sit better on a mailing list
>> than
>>>> in
>>>>> comments up on JIRA so here's an attempt at porting the chat
>> here.
>>>>>
>>>>> Background/Context: Bits of Hadoop and HBase had Apache HTrace
>>> trace
>>>>> points added. HTrace was formerly "incubating" at Apache but has
>>>> since
>>>>> been retired, moved to Apache Attic. HTrace and the efforts at
>>>>> instrumenting Hadoop wilted for want of attention/resourcing. Our
>>>> Todd
>>>>> Lipcon noticed that the HTrace instrumentation can add friction
>> on
>>>>> some code paths so can actually be harmful even when disabled.
>> The
>>>>> natural follow-on is that we should rip out tracings of a "dead"
>>>>> project. This then beggars the question, should something replace
>>> it
>>>>> and if so what? This is where HADOOP-15566 is at currently.
>>>>>
>>>>> HTrace took two or three runs, led by various Heros, at building
>> a
>>>>> trace lib for Hadoop (first). It was trying to build the trace
>>> lib, a
>>>>> store, and a visualizer. Always, it had a mechanism for dumping
>> the
>>>>> traces out to external systems for storage and viewing (e.g.
>>> Zipkin).
>>>>> HTrace started when there was little else but the, you guessed
>> it,
>>>>> Google paper that described the Dapper system they had
>> internally.
>>>>> Since then, the world of tracing has come on in leaps and bounds
>>> with
>>>>> healthy alternatives, communities, and even commercialization.
>>>>>
>>>>> If interested, take a read over HADOOP-15566. Will try and
>>> encourage
>>>>> participants to move the chat here.
>>>>>
>>>>> Thanks,
>>>>> St.Ack
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail:
>> common-dev-help@hadoop.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>>
>>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Stack <st...@duboce.net>.
On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
> What if someone built a HTrace facade for Zipkin / Brave?
I like the idea but taking a look, HTrace does static dispatch. I was
thinking that precludes our being able to do a facade. I would love to hear
otherwise.
Thanks,
S
> Hadoop, HBase,
> Phoenix, and other HTrace API users would still need to move away from
> embedding HTrace instrumentation points to whatever is the normal API of
> the accepted replacement, but such a facade would give you a drop in
> replacement requiring no code changes to currently shipping code lines, and
> some time to do a hopefully coordinated replacement involving all upstreams
> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
> that option and the impending incubation here at the ASF will make it quite
> attractive, I think.
>
>
> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>
> > On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > > Thanks for starting discussion, Stack.
> > >
> > > The ZipKin seems to be coming to the Apache Incubator. As Andrew
> > > Purtell said on HADOOP-15566, it would be good option since there is
> > > no problem about licenses.
> > > https://wiki.apache.org/incubator/ZipkinProposal
> > >
> > >
> > Yes. This is nice to see.
> >
> >
> >
> > > Stack, do you have any knowledge about differences between Zipkin and
> > > HTrace? Might measurable performance overhead be observed still in
> > > Zipkin?
> > >
> > >
> > I've not measured to see if disabled trace points are friction-free.
> > Perhaps someone else has?
> >
> >
> >
> > > To decrease the overhead, we need to do additional work like ftrace,
> > > well known dtrace implementation in Linux kernel. If I understand
> > > correctly, ftrace replace its function calls with NOP operations of
> > > CPU instruction when it is disabled. This ensures the lower overhead
> > > by the tracer. By replacing the function calls for tracing to JVM's
> > > NOP operation, can we achieve the minimum overhead?
> > >
> > >
> > That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
> > context, we should be able to get away with something less exotic.
> >
> > Thanks Tsuyoshi,
> > S
> >
> >
> >
> >
> > > Regards
> > > - Tsuyoshi
> > > On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
> wrote:
> > > >
> > > > Most of code coverage tools can instrument java classes without make
> > any
> > > > source code changes, but tracing distributed system is more involved
> > > because
> > > > code execution via network interactions are not easy to match up.
> > > > All interactions between sender and receiver have some form of
> session
> > id
> > > > or sequence id. Hadoop had some logic to assist the stitching of
> > > distributed
> > > > interactions together in clienttrace log. This information seems to
> > > have been
> > > > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
> to
> > > fill the void
> > > > left behind by clienttrace as a programmable API to send out useful
> > > tracing data for
> > > > downstream analytical program to visualize the interaction.
> > > >
> > > > Large companies have common practice to enforce logging the session
> id,
> > > and
> > > > write homebrew tools to stitch together debugging logic for a
> specific
> > > software.
> > > > There are also growing set of tools from Splunk or similar companies
> to
> > > write
> > > > analytical tools to stitch the views together. Hadoop does not seem
> to
> > > be on
> > > > top of the list for those company to implement the tracing because
> > Hadoop
> > > > networking layer is complex and changed more frequently than desired.
> > > >
> > > > If we go back to logging approach, instead of API approach, it will
> > help
> > > > someone to write the analytical program someday. The danger of
> logging
> > > > approach is that It is boring to write LOG.debug() everywhere, and we
> > > > often forgot about it, and log entries are removed.
> > > >
> > > > API approach can work, if real time interactive tracing can be done.
> > > > However, this is hard to realize in Hadoop because massive amount of
> > > > parallel data is difficult to aggregate at real time without hitting
> > > timeout.
> > > > It has a higher chance to require changes to network protocol that
> > might
> > > cause
> > > > more headache than it's worth. I am in favor of removing Htrace
> > support
> > > > and redo distributed tracing using logging approach.
> > > >
> > > > Regards,
> > > > Eric
> > > >
> > > > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> > > >
> > > > There is a healthy discussion going on over in HADOOP-15566 on
> > > tracing
> > > > in the Hadoop ecosystem. It would sit better on a mailing list
> than
> > > in
> > > > comments up on JIRA so here's an attempt at porting the chat
> here.
> > > >
> > > > Background/Context: Bits of Hadoop and HBase had Apache HTrace
> > trace
> > > > points added. HTrace was formerly "incubating" at Apache but has
> > > since
> > > > been retired, moved to Apache Attic. HTrace and the efforts at
> > > > instrumenting Hadoop wilted for want of attention/resourcing. Our
> > > Todd
> > > > Lipcon noticed that the HTrace instrumentation can add friction
> on
> > > > some code paths so can actually be harmful even when disabled.
> The
> > > > natural follow-on is that we should rip out tracings of a "dead"
> > > > project. This then beggars the question, should something replace
> > it
> > > > and if so what? This is where HADOOP-15566 is at currently.
> > > >
> > > > HTrace took two or three runs, led by various Heros, at building
> a
> > > > trace lib for Hadoop (first). It was trying to build the trace
> > lib, a
> > > > store, and a visualizer. Always, it had a mechanism for dumping
> the
> > > > traces out to external systems for storage and viewing (e.g.
> > Zipkin).
> > > > HTrace started when there was little else but the, you guessed
> it,
> > > > Google paper that described the Dapper system they had
> internally.
> > > > Since then, the world of tracing has come on in leaps and bounds
> > with
> > > > healthy alternatives, communities, and even commercialization.
> > > >
> > > > If interested, take a read over HADOOP-15566. Will try and
> > encourage
> > > > participants to move the chat here.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Stack <st...@duboce.net>.
On Tue, Aug 21, 2018 at 10:09 AM Andrew Purtell <ap...@apache.org> wrote:
> What if someone built a HTrace facade for Zipkin / Brave?
I like the idea but taking a look, HTrace does static dispatch. I was
thinking that precludes our being able to do a facade. I would love to hear
otherwise.
Thanks,
S
> Hadoop, HBase,
> Phoenix, and other HTrace API users would still need to move away from
> embedding HTrace instrumentation points to whatever is the normal API of
> the accepted replacement, but such a facade would give you a drop in
> replacement requiring no code changes to currently shipping code lines, and
> some time to do a hopefully coordinated replacement involving all upstreams
> and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
> that option and the impending incubation here at the ASF will make it quite
> attractive, I think.
>
>
> On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
>
> > On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> > > Thanks for starting discussion, Stack.
> > >
> > > The ZipKin seems to be coming to the Apache Incubator. As Andrew
> > > Purtell said on HADOOP-15566, it would be good option since there is
> > > no problem about licenses.
> > > https://wiki.apache.org/incubator/ZipkinProposal
> > >
> > >
> > Yes. This is nice to see.
> >
> >
> >
> > > Stack, do you have any knowledge about differences between Zipkin and
> > > HTrace? Might measurable performance overhead be observed still in
> > > Zipkin?
> > >
> > >
> > I've not measured to see if disabled trace points are friction-free.
> > Perhaps someone else has?
> >
> >
> >
> > > To decrease the overhead, we need to do additional work like ftrace,
> > > well known dtrace implementation in Linux kernel. If I understand
> > > correctly, ftrace replace its function calls with NOP operations of
> > > CPU instruction when it is disabled. This ensures the lower overhead
> > > by the tracer. By replacing the function calls for tracing to JVM's
> > > NOP operation, can we achieve the minimum overhead?
> > >
> > >
> > That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
> > context, we should be able to get away with something less exotic.
> >
> > Thanks Tsuyoshi,
> > S
> >
> >
> >
> >
> > > Regards
> > > - Tsuyoshi
> > > On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com>
> wrote:
> > > >
> > > > Most of code coverage tools can instrument java classes without make
> > any
> > > > source code changes, but tracing distributed system is more involved
> > > because
> > > > code execution via network interactions are not easy to match up.
> > > > All interactions between sender and receiver have some form of
> session
> > id
> > > > or sequence id. Hadoop had some logic to assist the stitching of
> > > distributed
> > > > interactions together in clienttrace log. This information seems to
> > > have been
> > > > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented
> to
> > > fill the void
> > > > left behind by clienttrace as a programmable API to send out useful
> > > tracing data for
> > > > downstream analytical program to visualize the interaction.
> > > >
> > > > Large companies have common practice to enforce logging the session
> id,
> > > and
> > > > write homebrew tools to stitch together debugging logic for a
> specific
> > > software.
> > > > There are also growing set of tools from Splunk or similar companies
> to
> > > write
> > > > analytical tools to stitch the views together. Hadoop does not seem
> to
> > > be on
> > > > top of the list for those company to implement the tracing because
> > Hadoop
> > > > networking layer is complex and changed more frequently than desired.
> > > >
> > > > If we go back to logging approach, instead of API approach, it will
> > help
> > > > someone to write the analytical program someday. The danger of
> logging
> > > > approach is that It is boring to write LOG.debug() everywhere, and we
> > > > often forgot about it, and log entries are removed.
> > > >
> > > > API approach can work, if real time interactive tracing can be done.
> > > > However, this is hard to realize in Hadoop because massive amount of
> > > > parallel data is difficult to aggregate at real time without hitting
> > > timeout.
> > > > It has a higher chance to require changes to network protocol that
> > might
> > > cause
> > > > more headache than it's worth. I am in favor of removing Htrace
> > support
> > > > and redo distributed tracing using logging approach.
> > > >
> > > > Regards,
> > > > Eric
> > > >
> > > > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> > > >
> > > > There is a healthy discussion going on over in HADOOP-15566 on
> > > tracing
> > > > in the Hadoop ecosystem. It would sit better on a mailing list
> than
> > > in
> > > > comments up on JIRA so here's an attempt at porting the chat
> here.
> > > >
> > > > Background/Context: Bits of Hadoop and HBase had Apache HTrace
> > trace
> > > > points added. HTrace was formerly "incubating" at Apache but has
> > > since
> > > > been retired, moved to Apache Attic. HTrace and the efforts at
> > > > instrumenting Hadoop wilted for want of attention/resourcing. Our
> > > Todd
> > > > Lipcon noticed that the HTrace instrumentation can add friction
> on
> > > > some code paths so can actually be harmful even when disabled.
> The
> > > > natural follow-on is that we should rip out tracings of a "dead"
> > > > project. This then beggars the question, should something replace
> > it
> > > > and if so what? This is where HADOOP-15566 is at currently.
> > > >
> > > > HTrace took two or three runs, led by various Heros, at building
> a
> > > > trace lib for Hadoop (first). It was trying to build the trace
> > lib, a
> > > > store, and a visualizer. Always, it had a mechanism for dumping
> the
> > > > traces out to external systems for storage and viewing (e.g.
> > Zipkin).
> > > > HTrace started when there was little else but the, you guessed
> it,
> > > > Google paper that described the Dapper system they had
> internally.
> > > > Since then, the world of tracing has come on in leaps and bounds
> > with
> > > > healthy alternatives, communities, and even commercialization.
> > > >
> > > > If interested, take a read over HADOOP-15566. Will try and
> > encourage
> > > > participants to move the chat here.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > >
> > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail:
> common-dev-help@hadoop.apache.org
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <ap...@apache.org>.
What if someone built a HTrace facade for Zipkin / Brave? Hadoop, HBase,
Phoenix, and other HTrace API users would still need to move away from
embedding HTrace instrumentation points to whatever is the normal API of
the accepted replacement, but such a facade would give you a drop in
replacement requiring no code changes to currently shipping code lines, and
some time to do a hopefully coordinated replacement involving all upstreams
and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
that option and the impending incubation here at the ASF will make it quite
attractive, I think.
On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> > Thanks for starting discussion, Stack.
> >
> > The ZipKin seems to be coming to the Apache Incubator. As Andrew
> > Purtell said on HADOOP-15566, it would be good option since there is
> > no problem about licenses.
> > https://wiki.apache.org/incubator/ZipkinProposal
> >
> >
> Yes. This is nice to see.
>
>
>
> > Stack, do you have any knowledge about differences between Zipkin and
> > HTrace? Might measurable performance overhead be observed still in
> > Zipkin?
> >
> >
> I've not measured to see if disabled trace points are friction-free.
> Perhaps someone else has?
>
>
>
> > To decrease the overhead, we need to do additional work like ftrace,
> > well known dtrace implementation in Linux kernel. If I understand
> > correctly, ftrace replace its function calls with NOP operations of
> > CPU instruction when it is disabled. This ensures the lower overhead
> > by the tracer. By replacing the function calls for tracing to JVM's
> > NOP operation, can we achieve the minimum overhead?
> >
> >
> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
> context, we should be able to get away with something less exotic.
>
> Thanks Tsuyoshi,
> S
>
>
>
>
> > Regards
> > - Tsuyoshi
> > On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com> wrote:
> > >
> > > Most of code coverage tools can instrument java classes without make
> any
> > > source code changes, but tracing distributed system is more involved
> > because
> > > code execution via network interactions are not easy to match up.
> > > All interactions between sender and receiver have some form of session
> id
> > > or sequence id. Hadoop had some logic to assist the stitching of
> > distributed
> > > interactions together in clienttrace log. This information seems to
> > have been
> > > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented to
> > fill the void
> > > left behind by clienttrace as a programmable API to send out useful
> > tracing data for
> > > downstream analytical program to visualize the interaction.
> > >
> > > Large companies have common practice to enforce logging the session id,
> > and
> > > write homebrew tools to stitch together debugging logic for a specific
> > software.
> > > There are also growing set of tools from Splunk or similar companies to
> > write
> > > analytical tools to stitch the views together. Hadoop does not seem to
> > be on
> > > top of the list for those company to implement the tracing because
> Hadoop
> > > networking layer is complex and changed more frequently than desired.
> > >
> > > If we go back to logging approach, instead of API approach, it will
> help
> > > someone to write the analytical program someday. The danger of logging
> > > approach is that It is boring to write LOG.debug() everywhere, and we
> > > often forgot about it, and log entries are removed.
> > >
> > > API approach can work, if real time interactive tracing can be done.
> > > However, this is hard to realize in Hadoop because massive amount of
> > > parallel data is difficult to aggregate at real time without hitting
> > timeout.
> > > It has a higher chance to require changes to network protocol that
> might
> > cause
> > > more headache than it's worth. I am in favor of removing Htrace
> support
> > > and redo distributed tracing using logging approach.
> > >
> > > Regards,
> > > Eric
> > >
> > > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> > >
> > > There is a healthy discussion going on over in HADOOP-15566 on
> > tracing
> > > in the Hadoop ecosystem. It would sit better on a mailing list than
> > in
> > > comments up on JIRA so here's an attempt at porting the chat here.
> > >
> > > Background/Context: Bits of Hadoop and HBase had Apache HTrace
> trace
> > > points added. HTrace was formerly "incubating" at Apache but has
> > since
> > > been retired, moved to Apache Attic. HTrace and the efforts at
> > > instrumenting Hadoop wilted for want of attention/resourcing. Our
> > Todd
> > > Lipcon noticed that the HTrace instrumentation can add friction on
> > > some code paths so can actually be harmful even when disabled. The
> > > natural follow-on is that we should rip out tracings of a "dead"
> > > project. This then beggars the question, should something replace
> it
> > > and if so what? This is where HADOOP-15566 is at currently.
> > >
> > > HTrace took two or three runs, led by various Heros, at building a
> > > trace lib for Hadoop (first). It was trying to build the trace
> lib, a
> > > store, and a visualizer. Always, it had a mechanism for dumping the
> > > traces out to external systems for storage and viewing (e.g.
> Zipkin).
> > > HTrace started when there was little else but the, you guessed it,
> > > Google paper that described the Dapper system they had internally.
> > > Since then, the world of tracing has come on in leaps and bounds
> with
> > > healthy alternatives, communities, and even commercialization.
> > >
> > > If interested, take a read over HADOOP-15566. Will try and
> encourage
> > > participants to move the chat here.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Andrew Purtell <ap...@apache.org>.
What if someone built a HTrace facade for Zipkin / Brave? Hadoop, HBase,
Phoenix, and other HTrace API users would still need to move away from
embedding HTrace instrumentation points to whatever is the normal API of
the accepted replacement, but such a facade would give you a drop in
replacement requiring no code changes to currently shipping code lines, and
some time to do a hopefully coordinated replacement involving all upstreams
and downstreams. Just a thought. Zipkin / Brave has widespread adoption of
that option and the impending incubation here at the ASF will make it quite
attractive, I think.
On Tue, Aug 21, 2018 at 7:50 AM Stack <st...@duboce.net> wrote:
> On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> > Thanks for starting discussion, Stack.
> >
> > The ZipKin seems to be coming to the Apache Incubator. As Andrew
> > Purtell said on HADOOP-15566, it would be good option since there is
> > no problem about licenses.
> > https://wiki.apache.org/incubator/ZipkinProposal
> >
> >
> Yes. This is nice to see.
>
>
>
> > Stack, do you have any knowledge about differences between Zipkin and
> > HTrace? Might measurable performance overhead be observed still in
> > Zipkin?
> >
> >
> I've not measured to see if disabled trace points are friction-free.
> Perhaps someone else has?
>
>
>
> > To decrease the overhead, we need to do additional work like ftrace,
> > well known dtrace implementation in Linux kernel. If I understand
> > correctly, ftrace replace its function calls with NOP operations of
> > CPU instruction when it is disabled. This ensures the lower overhead
> > by the tracer. By replacing the function calls for tracing to JVM's
> > NOP operation, can we achieve the minimum overhead?
> >
> >
> That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
> context, we should be able to get away with something less exotic.
>
> Thanks Tsuyoshi,
> S
>
>
>
>
> > Regards
> > - Tsuyoshi
> > On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com> wrote:
> > >
> > > Most of code coverage tools can instrument java classes without make
> any
> > > source code changes, but tracing distributed system is more involved
> > because
> > > code execution via network interactions are not easy to match up.
> > > All interactions between sender and receiver have some form of session
> id
> > > or sequence id. Hadoop had some logic to assist the stitching of
> > distributed
> > > interactions together in clienttrace log. This information seems to
> > have been
> > > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented to
> > fill the void
> > > left behind by clienttrace as a programmable API to send out useful
> > tracing data for
> > > downstream analytical program to visualize the interaction.
> > >
> > > Large companies have common practice to enforce logging the session id,
> > and
> > > write homebrew tools to stitch together debugging logic for a specific
> > software.
> > > There are also growing set of tools from Splunk or similar companies to
> > write
> > > analytical tools to stitch the views together. Hadoop does not seem to
> > be on
> > > top of the list for those company to implement the tracing because
> Hadoop
> > > networking layer is complex and changed more frequently than desired.
> > >
> > > If we go back to logging approach, instead of API approach, it will
> help
> > > someone to write the analytical program someday. The danger of logging
> > > approach is that It is boring to write LOG.debug() everywhere, and we
> > > often forgot about it, and log entries are removed.
> > >
> > > API approach can work, if real time interactive tracing can be done.
> > > However, this is hard to realize in Hadoop because massive amount of
> > > parallel data is difficult to aggregate at real time without hitting
> > timeout.
> > > It has a higher chance to require changes to network protocol that
> might
> > cause
> > > more headache than it's worth. I am in favor of removing Htrace
> support
> > > and redo distributed tracing using logging approach.
> > >
> > > Regards,
> > > Eric
> > >
> > > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> > >
> > > There is a healthy discussion going on over in HADOOP-15566 on
> > tracing
> > > in the Hadoop ecosystem. It would sit better on a mailing list than
> > in
> > > comments up on JIRA so here's an attempt at porting the chat here.
> > >
> > > Background/Context: Bits of Hadoop and HBase had Apache HTrace
> trace
> > > points added. HTrace was formerly "incubating" at Apache but has
> > since
> > > been retired, moved to Apache Attic. HTrace and the efforts at
> > > instrumenting Hadoop wilted for want of attention/resourcing. Our
> > Todd
> > > Lipcon noticed that the HTrace instrumentation can add friction on
> > > some code paths so can actually be harmful even when disabled. The
> > > natural follow-on is that we should rip out tracings of a "dead"
> > > project. This then beggars the question, should something replace
> it
> > > and if so what? This is where HADOOP-15566 is at currently.
> > >
> > > HTrace took two or three runs, led by various Heros, at building a
> > > trace lib for Hadoop (first). It was trying to build the trace
> lib, a
> > > store, and a visualizer. Always, it had a mechanism for dumping the
> > > traces out to external systems for storage and viewing (e.g.
> Zipkin).
> > > HTrace started when there was little else but the, you guessed it,
> > > Google paper that described the Dapper system they had internally.
> > > Since then, the world of tracing has come on in leaps and bounds
> with
> > > healthy alternatives, communities, and even commercialization.
> > >
> > > If interested, take a read over HADOOP-15566. Will try and
> encourage
> > > participants to move the chat here.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Stack <st...@duboce.net>.
On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Thanks for starting discussion, Stack.
>
> The ZipKin seems to be coming to the Apache Incubator. As Andrew
> Purtell said on HADOOP-15566, it would be good option since there is
> no problem about licenses.
> https://wiki.apache.org/incubator/ZipkinProposal
>
>
Yes. This is nice to see.
> Stack, do you have any knowledge about differences between Zipkin and
> HTrace? Might measurable performance overhead be observed still in
> Zipkin?
>
>
I've not measured to see if disabled trace points are friction-free.
Perhaps someone else has?
> To decrease the overhead, we need to do additional work like ftrace,
> well known dtrace implementation in Linux kernel. If I understand
> correctly, ftrace replace its function calls with NOP operations of
> CPU instruction when it is disabled. This ensures the lower overhead
> by the tracer. By replacing the function calls for tracing to JVM's
> NOP operation, can we achieve the minimum overhead?
>
>
That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
context, we should be able to get away with something less exotic.
Thanks Tsuyoshi,
S
> Regards
> - Tsuyoshi
> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com> wrote:
> >
> > Most of code coverage tools can instrument java classes without make any
> > source code changes, but tracing distributed system is more involved
> because
> > code execution via network interactions are not easy to match up.
> > All interactions between sender and receiver have some form of session id
> > or sequence id. Hadoop had some logic to assist the stitching of
> distributed
> > interactions together in clienttrace log. This information seems to
> have been
> > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented to
> fill the void
> > left behind by clienttrace as a programmable API to send out useful
> tracing data for
> > downstream analytical program to visualize the interaction.
> >
> > Large companies have common practice to enforce logging the session id,
> and
> > write homebrew tools to stitch together debugging logic for a specific
> software.
> > There are also growing set of tools from Splunk or similar companies to
> write
> > analytical tools to stitch the views together. Hadoop does not seem to
> be on
> > top of the list for those company to implement the tracing because Hadoop
> > networking layer is complex and changed more frequently than desired.
> >
> > If we go back to logging approach, instead of API approach, it will help
> > someone to write the analytical program someday. The danger of logging
> > approach is that It is boring to write LOG.debug() everywhere, and we
> > often forgot about it, and log entries are removed.
> >
> > API approach can work, if real time interactive tracing can be done.
> > However, this is hard to realize in Hadoop because massive amount of
> > parallel data is difficult to aggregate at real time without hitting
> timeout.
> > It has a higher chance to require changes to network protocol that might
> cause
> > more headache than it's worth. I am in favor of removing Htrace support
> > and redo distributed tracing using logging approach.
> >
> > Regards,
> > Eric
> >
> > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> >
> > There is a healthy discussion going on over in HADOOP-15566 on
> tracing
> > in the Hadoop ecosystem. It would sit better on a mailing list than
> in
> > comments up on JIRA so here's an attempt at porting the chat here.
> >
> > Background/Context: Bits of Hadoop and HBase had Apache HTrace trace
> > points added. HTrace was formerly "incubating" at Apache but has
> since
> > been retired, moved to Apache Attic. HTrace and the efforts at
> > instrumenting Hadoop wilted for want of attention/resourcing. Our
> Todd
> > Lipcon noticed that the HTrace instrumentation can add friction on
> > some code paths so can actually be harmful even when disabled. The
> > natural follow-on is that we should rip out tracings of a "dead"
> > project. This then beggars the question, should something replace it
> > and if so what? This is where HADOOP-15566 is at currently.
> >
> > HTrace took two or three runs, led by various Heros, at building a
> > trace lib for Hadoop (first). It was trying to build the trace lib, a
> > store, and a visualizer. Always, it had a mechanism for dumping the
> > traces out to external systems for storage and viewing (e.g. Zipkin).
> > HTrace started when there was little else but the, you guessed it,
> > Google paper that described the Dapper system they had internally.
> > Since then, the world of tracing has come on in leaps and bounds with
> > healthy alternatives, communities, and even commercialization.
> >
> > If interested, take a read over HADOOP-15566. Will try and encourage
> > participants to move the chat here.
> >
> > Thanks,
> > St.Ack
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
Re: [DISCUSS] Tracing in the Hadoop ecosystem
Posted by Stack <st...@duboce.net>.
On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Thanks for starting discussion, Stack.
>
> The ZipKin seems to be coming to the Apache Incubator. As Andrew
> Purtell said on HADOOP-15566, it would be good option since there is
> no problem about licenses.
> https://wiki.apache.org/incubator/ZipkinProposal
>
>
Yes. This is nice to see.
> Stack, do you have any knowledge about differences between Zipkin and
> HTrace? Might measurable performance overhead be observed still in
> Zipkin?
>
>
I've not measured to see if disabled trace points are friction-free.
Perhaps someone else has?
> To decrease the overhead, we need to do additional work like ftrace,
> well known dtrace implementation in Linux kernel. If I understand
> correctly, ftrace replace its function calls with NOP operations of
> CPU instruction when it is disabled. This ensures the lower overhead
> by the tracer. By replacing the function calls for tracing to JVM's
> NOP operation, can we achieve the minimum overhead?
>
>
That'd be ideal. Makes sense inside the kernel. But up in our sloppy java
context, we should be able to get away with something less exotic.
Thanks Tsuyoshi,
S
> Regards
> - Tsuyoshi
> On Tue, Jul 31, 2018 at 9:59 AM Eric Yang <ey...@hortonworks.com> wrote:
> >
> > Most of code coverage tools can instrument java classes without make any
> > source code changes, but tracing distributed system is more involved
> because
> > code execution via network interactions are not easy to match up.
> > All interactions between sender and receiver have some form of session id
> > or sequence id. Hadoop had some logic to assist the stitching of
> distributed
> > interactions together in clienttrace log. This information seems to
> have been
> > lost in the last 5-6 years of Hadoop evolutions. Htrace is invented to
> fill the void
> > left behind by clienttrace as a programmable API to send out useful
> tracing data for
> > downstream analytical program to visualize the interaction.
> >
> > Large companies have common practice to enforce logging the session id,
> and
> > write homebrew tools to stitch together debugging logic for a specific
> software.
> > There are also growing set of tools from Splunk or similar companies to
> write
> > analytical tools to stitch the views together. Hadoop does not seem to
> be on
> > top of the list for those company to implement the tracing because Hadoop
> > networking layer is complex and changed more frequently than desired.
> >
> > If we go back to logging approach, instead of API approach, it will help
> > someone to write the analytical program someday. The danger of logging
> > approach is that It is boring to write LOG.debug() everywhere, and we
> > often forgot about it, and log entries are removed.
> >
> > API approach can work, if real time interactive tracing can be done.
> > However, this is hard to realize in Hadoop because massive amount of
> > parallel data is difficult to aggregate at real time without hitting
> timeout.
> > It has a higher chance to require changes to network protocol that might
> cause
> > more headache than it's worth. I am in favor of removing Htrace support
> > and redo distributed tracing using logging approach.
> >
> > Regards,
> > Eric
> >
> > On 7/30/18, 3:06 PM, "Stack" <st...@duboce.net> wrote:
> >
> > There is a healthy discussion going on over in HADOOP-15566 on
> tracing
> > in the Hadoop ecosystem. It would sit better on a mailing list than
> in
> > comments up on JIRA so here's an attempt at porting the chat here.
> >
> > Background/Context: Bits of Hadoop and HBase had Apache HTrace trace
> > points added. HTrace was formerly "incubating" at Apache but has
> since
> > been retired, moved to Apache Attic. HTrace and the efforts at
> > instrumenting Hadoop wilted for want of attention/resourcing. Our
> Todd
> > Lipcon noticed that the HTrace instrumentation can add friction on
> > some code paths so can actually be harmful even when disabled. The
> > natural follow-on is that we should rip out tracings of a "dead"
> > project. This then beggars the question, should something replace it
> > and if so what? This is where HADOOP-15566 is at currently.
> >
> > HTrace took two or three runs, led by various Heros, at building a
> > trace lib for Hadoop (first). It was trying to build the trace lib, a
> > store, and a visualizer. Always, it had a mechanism for dumping the
> > traces out to external systems for storage and viewing (e.g. Zipkin).
> > HTrace started when there was little else but the, you guessed it,
> > Google paper that described the Dapper system they had internally.
> > Since then, the world of tracing has come on in leaps and bounds with
> > healthy alternatives, communities, and even commercialization.
> >
> > If interested, take a read over HADOOP-15566. Will try and encourage
> > participants to move the chat here.
> >
> > Thanks,
> > St.Ack
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
>