You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Sean Busbey <bu...@apache.org> on 2018/02/27 16:23:19 UTC
[DISCUSS] tracing framework updates
More things we should get ahead of for Accumulo 2.0.0: distributed tracing.
Right now we have an awkward situation wrt HTrace support. We're using and shipping htrace 3.1. It works okay for our internal uses, afaict.
Hadoop 2.6 ships and uses HTrace 3.0. I believe this does not work with 3.1.
Hadoop 2.7 ships and uses HTrace 3.1, which means we can trace things down through the hdfs layer, though I've never managed to get anything useful out of trying to do so.
Hadoop 2.8+ ships and uses HTrace 4.0, which isn't compatible with 3.1.
Hadoop 2.9+ and 3.0+ ship and use HTrace 4.1, it doesn't work with 3.1 and it's not clear to me if it works with 4.0.
I *think* this means that we can only do tracing down into a single minor release line of HDFS.
Additionally, as of their January podling report[1] it looks like HTrace is about ready to shut down as a project. In theory it could continue on github or something after exiting the ASF incubator, but I don't see why we'd expect an increase in activity; the project's been pretty idle for at least a year.
I really don't want to go back to maintaining cloudtrace. So what should we do? Zipkin has some decent uptake[2]; they're still making regular releases. Maybe just adopt OpenTracing[3] as a way to hedge against having to do his again?
YCSB is the Accumulo downstream with tracing that I'm most familiar with. It relies on HTrace 4.1. I'm probably going to push that community to abandon HTrace soon. (I was the one that pushed it to adopt HTrace in the first place). I'd like to at least get tracing from YCSB through Accumulo in the process.
[1]: https://s.apache.org/uDHN
[2]: https://zipkin.io/
[3]: http://opentracing.io/documentation/
Re: [DISCUSS] tracing framework updates
Posted by Christopher <ct...@apache.org>.
I wonder if it would be better to just rip out tracing in 2.0. I know it's
a big leap... but I really don't know anybody using it regularly, and given
the potential compatibility problems, potential impact of migrating, and
general lack of care from the HTrace project for API compatibility, as well
as a myriad of various tracing-related bugs that are hard to fix
(ACCUMULO-4415 off the top of my head, but I think there's also generally a
problem of unrooted traces due to trace-unaware executors and similar
problems).
Ripping it out of 2.0 could give us some breathing room to do some internal
modularization/refactoring/cleaning up code debt, before potentially
reintroducing it more carefully in the future.
I'm not saying I like this option... but it might be worth considering.
On Tue, Feb 27, 2018 at 1:09 PM Ed Coleman <de...@etcoleman.com> wrote:
> For general discussion - Facebook recently (Oct 28, 2017) published a
> paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
> System (
> https://research.fb.com/publications/canopy-end-to-end-performance-tracing-at-scale/
> )
>
> As a bonus, they referenced Accumulo and HTrace in section 2.2
>
> "Mismatched models affected compatibility between mixed system versions;
> e.g. Accumulo and Hadoop were impacted by the “continued lack of concern in
> the HTrace project around tracing during upgrades”
>
>
> -----Original Message-----
> From: Tony Kurc [mailto:tkurc@apache.org]
> Sent: Tuesday, February 27, 2018 12:57 PM
> To: dev@accumulo.apache.org
> Subject: Re: [DISCUSS] tracing framework updates
>
> I have some experience with opentracing, and it definitely seems
> promising, however, potentially promising in the same way htrace was...
> That being said, I did a cursory thought exercise of what it would take to
> do a swap of the current tracing in accumulo to opentracing, and I didn't
> come across any hard problems, meaning it could be a fairly straightforward
> refactor. I was hoping to explore the community a bit more at some upcoming
> conferences
>
> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>
> >
> >
> > On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
> > > I didn't realize HTrace was struggling in incubation. Maybe some of
> > > us
> > can
> > > start participating? The project did start within Accumulo, after all.
> > What
> > > does it need? I also wouldn't want to go back to maintaining
> cloudtrace.
> > >
> >
> > I suspect it's too late for HTrace. The last commit to the main
> > development branch was May 2017. They had a decent run of activity in
> > 2015 and an almost-resurgence in 2016, but they never really got
> > enough community traction to survive the normal ebb and flow of
> > contributor involvement.
> >
> > They need the things any project needs to be sustainable: regular
> > release cadences, a responsive contribution process, and folks to do
> > the long slog of building interest via e.g. production adoption.
> >
> > > I'm unfamiliar with OpenTracing, but it was my understanding that
> > > Zipkin was more of a tracing sink, than an instrumentation API.
> > > HTrace is
> > actually
> > > listed as an instrumentation library for Zipkin (among others).
> > >
> >
> > I think the key is that for a instrumentation library to get adoption
> > it needs a good sink that provides utility to operators looking to
> > diagnose problems. It took too long for HTrace to provide any tooling
> > that could help with even simple performance profiling. Maybe hooking
> > it into Zipkin would get around that. Personally, I never managed to
> > get the two to actually work together.
> >
> > My listing Zipkin as an option merely reflects my prioritization of
> > practical impact of whatever we go to. I don't want to adopt some
> > blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
> > a zipkin-sink compatible runtime.
> >
> > There's a whole community that just does distributed monitoring, maybe
> > someone has time to survey some spaces and see if OpenTracing has any
> legs.
> >
>
>
Re: [DISCUSS] tracing framework updates
Posted by Christopher <ct...@apache.org>.
I agree that it has value, and would also be disappointed ripping it out.
However, unless the interest in keeping it is accompanied by interest in
maintaining it, I don't think we have much choice. Eventually, we're going
to have to make hard decisions about what things to keep, and what things
to drop. If we fail to make these hard decisions, these lesser maintained
features will (at some point) become a drag on the entire project.
On Tue, Feb 27, 2018 at 6:53 PM Tony Kurc <tr...@gmail.com> wrote:
> I'd personally be disappointed to see it removed. There is a bit of a
> learning curve and startup cost to use it now, but when diagnosing major
> challenges, it has been an invaluable capability.
>
> On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
>
> Wow... that's, erm, quite the paper. Nothing like taking some pot-shots at
> another software project and quoting folks out of context.
>
> Does it help to break down the problem some more?
>
> * Is Accumulo getting benefit from tracing its library?
> * Is Accumulo getting benefit from tracing context including HDFS calls?
>
> I feel like it is a nice tool to have in your toolbelt (having used it
> successfully in the past), but I wonder if it's the most effective thing to
> keep inside of Accumulo. Specifically, would it be better to just pull this
> out of Accumulo outright?
>
> I don't think I have an opinion yet.
>
>
> On 2/27/18 1:08 PM, Ed Coleman wrote:
>
> > For general discussion - Facebook recently (Oct 28, 2017) published a
> > paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
> > System (https://research.fb.com/publications/canopy-end-to-end-
> > performance-tracing-at-scale/)
> >
> > As a bonus, they referenced Accumulo and HTrace in section 2.2
> >
> > "Mismatched models affected compatibility between mixed system versions;
> > e.g. Accumulo and Hadoop were impacted by the “continued lack of concern
> in
> > the HTrace project around tracing during upgrades”
> >
> >
> > -----Original Message-----
> > From: Tony Kurc [mailto:tkurc@apache.org]
> > Sent: Tuesday, February 27, 2018 12:57 PM
> > To: dev@accumulo.apache.org
> > Subject: Re: [DISCUSS] tracing framework updates
> >
> > I have some experience with opentracing, and it definitely seems
> > promising, however, potentially promising in the same way htrace was...
> > That being said, I did a cursory thought exercise of what it would take
> to
> > do a swap of the current tracing in accumulo to opentracing, and I didn't
> > come across any hard problems, meaning it could be a fairly
> straightforward
> > refactor. I was hoping to explore the community a bit more at some
> upcoming
> > conferences
> >
> > On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
> >
> >
> >>
> >> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
> >>
> >>> I didn't realize HTrace was struggling in incubation. Maybe some of
> >>> us
> >>>
> >> can
> >>
> >>> start participating? The project did start within Accumulo, after all.
> >>>
> >> What
> >>
> >>> does it need? I also wouldn't want to go back to maintaining
> cloudtrace.
> >>>
> >>>
> >> I suspect it's too late for HTrace. The last commit to the main
> >> development branch was May 2017. They had a decent run of activity in
> >> 2015 and an almost-resurgence in 2016, but they never really got
> >> enough community traction to survive the normal ebb and flow of
> >> contributor involvement.
> >>
> >> They need the things any project needs to be sustainable: regular
> >> release cadences, a responsive contribution process, and folks to do
> >> the long slog of building interest via e.g. production adoption.
> >>
> >> I'm unfamiliar with OpenTracing, but it was my understanding that
> >>> Zipkin was more of a tracing sink, than an instrumentation API.
> >>> HTrace is
> >>>
> >> actually
> >>
> >>> listed as an instrumentation library for Zipkin (among others).
> >>>
> >>>
> >> I think the key is that for a instrumentation library to get adoption
> >> it needs a good sink that provides utility to operators looking to
> >> diagnose problems. It took too long for HTrace to provide any tooling
> >> that could help with even simple performance profiling. Maybe hooking
> >> it into Zipkin would get around that. Personally, I never managed to
> >> get the two to actually work together.
> >>
> >> My listing Zipkin as an option merely reflects my prioritization of
> >> practical impact of whatever we go to. I don't want to adopt some
> >> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
> >> a zipkin-sink compatible runtime.
> >>
> >> There's a whole community that just does distributed monitoring, maybe
> >> someone has time to survey some spaces and see if OpenTracing has any
> >> legs.
> >>
> >>
> >
>
Re: [DISCUSS] tracing framework updates
Posted by Tony Kurc <tr...@gmail.com>.
Josh,
In theory, I'd be most happy if it were possible to have both sides of the
accumulo sandwich (one side being "the application", the other being
HDFS/zookeeper/etc.) instrumented and tracing in the same way to get a
comprehensive view. I just don't know how realistic that is.
On Wed, Feb 28, 2018 at 10:02 AM, Josh Elser <el...@apache.org> wrote:
> Thanks for letting us know, Tony. I can totally understand how the
> server-side tracing (and collapsing it can do) would be super-helpful in
> figuring out what's happening.
>
> I read that as one reason for simply not trying to get HDFS and Accumulo
> re-sync'ed. I think we have value in leaving what we presently have in
> Accumulo now over removing it completely.
>
>
> On 2/27/18 8:50 PM, Tony Kurc wrote:
>
>> Josh,
>> It was exclusively the first - using the traces in the server-side code.
>> The most common case is "I have a scan which is much slower than
>> expected",
>> and couldn't figure out why. I'm trying to think of alternative approaches
>> to using the traces, and honestly, doing a bunch of log aggregation is the
>> alternative I'd have to fall back to, and in some cases recompiling parts
>> of accumulo with new log messages in place.
>>
>>
>> Tony
>>
>> On Tue, Feb 27, 2018 at 7:18 PM, Josh Elser <el...@apache.org> wrote:
>>
>> Oh, that's a pleasant surprise to hear, actually.
>>>
>>> Anything you can share with the class, Tony? Would love to hear (even if
>>> brief) how it was used and benefited you.
>>>
>>> Specifically, I'm curious if...
>>>
>>> * You looked at traces from our server-side instrumented code
>>> * You instrumented your own code outside of Accumulo and used Accumulo as
>>> the backing store
>>> * You instrumented code inside/outside Accumulo and benefited from the
>>> server-side instrumentation (e.g. your code's spans collapsing with the
>>> server's spans)
>>>
>>>
>>> On 2/27/18 6:52 PM, Tony Kurc wrote:
>>>
>>> I'd personally be disappointed to see it removed. There is a bit of a
>>>> learning curve and startup cost to use it now, but when diagnosing major
>>>> challenges, it has been an invaluable capability.
>>>>
>>>> On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
>>>>
>>>> Wow... that's, erm, quite the paper. Nothing like taking some pot-shots
>>>> at
>>>> another software project and quoting folks out of context.
>>>>
>>>> Does it help to break down the problem some more?
>>>>
>>>> * Is Accumulo getting benefit from tracing its library?
>>>> * Is Accumulo getting benefit from tracing context including HDFS calls?
>>>>
>>>> I feel like it is a nice tool to have in your toolbelt (having used it
>>>> successfully in the past), but I wonder if it's the most effective thing
>>>> to
>>>> keep inside of Accumulo. Specifically, would it be better to just pull
>>>> this
>>>> out of Accumulo outright?
>>>>
>>>> I don't think I have an opinion yet.
>>>>
>>>>
>>>> On 2/27/18 1:08 PM, Ed Coleman wrote:
>>>>
>>>> For general discussion - Facebook recently (Oct 28, 2017) published a
>>>>
>>>>> paper on tracing: Canopy: An End-to-End Performance Tracing and
>>>>> Analysis
>>>>> System (https://research.fb.com/publications/canopy-end-to-end-
>>>>> performance-tracing-at-scale/)
>>>>>
>>>>> As a bonus, they referenced Accumulo and HTrace in section 2.2
>>>>>
>>>>> "Mismatched models affected compatibility between mixed system
>>>>> versions;
>>>>> e.g. Accumulo and Hadoop were impacted by the “continued lack of
>>>>> concern
>>>>> in
>>>>> the HTrace project around tracing during upgrades”
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Tony Kurc [mailto:tkurc@apache.org]
>>>>> Sent: Tuesday, February 27, 2018 12:57 PM
>>>>> To: dev@accumulo.apache.org
>>>>> Subject: Re: [DISCUSS] tracing framework updates
>>>>>
>>>>> I have some experience with opentracing, and it definitely seems
>>>>> promising, however, potentially promising in the same way htrace was...
>>>>> That being said, I did a cursory thought exercise of what it would take
>>>>> to
>>>>> do a swap of the current tracing in accumulo to opentracing, and I
>>>>> didn't
>>>>> come across any hard problems, meaning it could be a fairly
>>>>> straightforward
>>>>> refactor. I was hoping to explore the community a bit more at some
>>>>> upcoming
>>>>> conferences
>>>>>
>>>>> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>>>>>
>>>>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>>>>>
>>>>>>> us
>>>>>>>
>>>>>>> can
>>>>>>>
>>>>>>
>>>>>> start participating? The project did start within Accumulo, after all.
>>>>>>
>>>>>>>
>>>>>>> What
>>>>>>>
>>>>>>
>>>>>> does it need? I also wouldn't want to go back to maintaining
>>>>>> cloudtrace.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I suspect it's too late for HTrace. The last commit to the main
>>>>>>>
>>>>>> development branch was May 2017. They had a decent run of activity in
>>>>>> 2015 and an almost-resurgence in 2016, but they never really got
>>>>>> enough community traction to survive the normal ebb and flow of
>>>>>> contributor involvement.
>>>>>>
>>>>>> They need the things any project needs to be sustainable: regular
>>>>>> release cadences, a responsive contribution process, and folks to do
>>>>>> the long slog of building interest via e.g. production adoption.
>>>>>>
>>>>>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>>>>>
>>>>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>>>>>> HTrace is
>>>>>>>
>>>>>>> actually
>>>>>>>
>>>>>>
>>>>>> listed as an instrumentation library for Zipkin (among others).
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think the key is that for a instrumentation library to get adoption
>>>>>>>
>>>>>> it needs a good sink that provides utility to operators looking to
>>>>>> diagnose problems. It took too long for HTrace to provide any tooling
>>>>>> that could help with even simple performance profiling. Maybe hooking
>>>>>> it into Zipkin would get around that. Personally, I never managed to
>>>>>> get the two to actually work together.
>>>>>>
>>>>>> My listing Zipkin as an option merely reflects my prioritization of
>>>>>> practical impact of whatever we go to. I don't want to adopt some
>>>>>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>>>>>> a zipkin-sink compatible runtime.
>>>>>>
>>>>>> There's a whole community that just does distributed monitoring, maybe
>>>>>> someone has time to survey some spaces and see if OpenTracing has any
>>>>>> legs.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
Re: [DISCUSS] tracing framework updates
Posted by Josh Elser <el...@apache.org>.
Thanks for letting us know, Tony. I can totally understand how the
server-side tracing (and collapsing it can do) would be super-helpful in
figuring out what's happening.
I read that as one reason for simply not trying to get HDFS and Accumulo
re-sync'ed. I think we have value in leaving what we presently have in
Accumulo now over removing it completely.
On 2/27/18 8:50 PM, Tony Kurc wrote:
> Josh,
> It was exclusively the first - using the traces in the server-side code.
> The most common case is "I have a scan which is much slower than expected",
> and couldn't figure out why. I'm trying to think of alternative approaches
> to using the traces, and honestly, doing a bunch of log aggregation is the
> alternative I'd have to fall back to, and in some cases recompiling parts
> of accumulo with new log messages in place.
>
>
> Tony
>
> On Tue, Feb 27, 2018 at 7:18 PM, Josh Elser <el...@apache.org> wrote:
>
>> Oh, that's a pleasant surprise to hear, actually.
>>
>> Anything you can share with the class, Tony? Would love to hear (even if
>> brief) how it was used and benefited you.
>>
>> Specifically, I'm curious if...
>>
>> * You looked at traces from our server-side instrumented code
>> * You instrumented your own code outside of Accumulo and used Accumulo as
>> the backing store
>> * You instrumented code inside/outside Accumulo and benefited from the
>> server-side instrumentation (e.g. your code's spans collapsing with the
>> server's spans)
>>
>>
>> On 2/27/18 6:52 PM, Tony Kurc wrote:
>>
>>> I'd personally be disappointed to see it removed. There is a bit of a
>>> learning curve and startup cost to use it now, but when diagnosing major
>>> challenges, it has been an invaluable capability.
>>>
>>> On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
>>>
>>> Wow... that's, erm, quite the paper. Nothing like taking some pot-shots at
>>> another software project and quoting folks out of context.
>>>
>>> Does it help to break down the problem some more?
>>>
>>> * Is Accumulo getting benefit from tracing its library?
>>> * Is Accumulo getting benefit from tracing context including HDFS calls?
>>>
>>> I feel like it is a nice tool to have in your toolbelt (having used it
>>> successfully in the past), but I wonder if it's the most effective thing
>>> to
>>> keep inside of Accumulo. Specifically, would it be better to just pull
>>> this
>>> out of Accumulo outright?
>>>
>>> I don't think I have an opinion yet.
>>>
>>>
>>> On 2/27/18 1:08 PM, Ed Coleman wrote:
>>>
>>> For general discussion - Facebook recently (Oct 28, 2017) published a
>>>> paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
>>>> System (https://research.fb.com/publications/canopy-end-to-end-
>>>> performance-tracing-at-scale/)
>>>>
>>>> As a bonus, they referenced Accumulo and HTrace in section 2.2
>>>>
>>>> "Mismatched models affected compatibility between mixed system versions;
>>>> e.g. Accumulo and Hadoop were impacted by the “continued lack of concern
>>>> in
>>>> the HTrace project around tracing during upgrades”
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Tony Kurc [mailto:tkurc@apache.org]
>>>> Sent: Tuesday, February 27, 2018 12:57 PM
>>>> To: dev@accumulo.apache.org
>>>> Subject: Re: [DISCUSS] tracing framework updates
>>>>
>>>> I have some experience with opentracing, and it definitely seems
>>>> promising, however, potentially promising in the same way htrace was...
>>>> That being said, I did a cursory thought exercise of what it would take
>>>> to
>>>> do a swap of the current tracing in accumulo to opentracing, and I didn't
>>>> come across any hard problems, meaning it could be a fairly
>>>> straightforward
>>>> refactor. I was hoping to explore the community a bit more at some
>>>> upcoming
>>>> conferences
>>>>
>>>> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>>>>
>>>>
>>>>
>>>>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>>>>
>>>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>>>>> us
>>>>>>
>>>>>> can
>>>>>
>>>>> start participating? The project did start within Accumulo, after all.
>>>>>>
>>>>>> What
>>>>>
>>>>> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>>>>>>
>>>>>>
>>>>>> I suspect it's too late for HTrace. The last commit to the main
>>>>> development branch was May 2017. They had a decent run of activity in
>>>>> 2015 and an almost-resurgence in 2016, but they never really got
>>>>> enough community traction to survive the normal ebb and flow of
>>>>> contributor involvement.
>>>>>
>>>>> They need the things any project needs to be sustainable: regular
>>>>> release cadences, a responsive contribution process, and folks to do
>>>>> the long slog of building interest via e.g. production adoption.
>>>>>
>>>>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>>>>
>>>>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>>>>> HTrace is
>>>>>>
>>>>>> actually
>>>>>
>>>>> listed as an instrumentation library for Zipkin (among others).
>>>>>>
>>>>>>
>>>>>> I think the key is that for a instrumentation library to get adoption
>>>>> it needs a good sink that provides utility to operators looking to
>>>>> diagnose problems. It took too long for HTrace to provide any tooling
>>>>> that could help with even simple performance profiling. Maybe hooking
>>>>> it into Zipkin would get around that. Personally, I never managed to
>>>>> get the two to actually work together.
>>>>>
>>>>> My listing Zipkin as an option merely reflects my prioritization of
>>>>> practical impact of whatever we go to. I don't want to adopt some
>>>>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>>>>> a zipkin-sink compatible runtime.
>>>>>
>>>>> There's a whole community that just does distributed monitoring, maybe
>>>>> someone has time to survey some spaces and see if OpenTracing has any
>>>>> legs.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>
Re: [DISCUSS] tracing framework updates
Posted by Tony Kurc <tr...@gmail.com>.
Josh,
It was exclusively the first - using the traces in the server-side code.
The most common case is "I have a scan which is much slower than expected",
and couldn't figure out why. I'm trying to think of alternative approaches
to using the traces, and honestly, doing a bunch of log aggregation is the
alternative I'd have to fall back to, and in some cases recompiling parts
of accumulo with new log messages in place.
Tony
On Tue, Feb 27, 2018 at 7:18 PM, Josh Elser <el...@apache.org> wrote:
> Oh, that's a pleasant surprise to hear, actually.
>
> Anything you can share with the class, Tony? Would love to hear (even if
> brief) how it was used and benefited you.
>
> Specifically, I'm curious if...
>
> * You looked at traces from our server-side instrumented code
> * You instrumented your own code outside of Accumulo and used Accumulo as
> the backing store
> * You instrumented code inside/outside Accumulo and benefited from the
> server-side instrumentation (e.g. your code's spans collapsing with the
> server's spans)
>
>
> On 2/27/18 6:52 PM, Tony Kurc wrote:
>
>> I'd personally be disappointed to see it removed. There is a bit of a
>> learning curve and startup cost to use it now, but when diagnosing major
>> challenges, it has been an invaluable capability.
>>
>> On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
>>
>> Wow... that's, erm, quite the paper. Nothing like taking some pot-shots at
>> another software project and quoting folks out of context.
>>
>> Does it help to break down the problem some more?
>>
>> * Is Accumulo getting benefit from tracing its library?
>> * Is Accumulo getting benefit from tracing context including HDFS calls?
>>
>> I feel like it is a nice tool to have in your toolbelt (having used it
>> successfully in the past), but I wonder if it's the most effective thing
>> to
>> keep inside of Accumulo. Specifically, would it be better to just pull
>> this
>> out of Accumulo outright?
>>
>> I don't think I have an opinion yet.
>>
>>
>> On 2/27/18 1:08 PM, Ed Coleman wrote:
>>
>> For general discussion - Facebook recently (Oct 28, 2017) published a
>>> paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
>>> System (https://research.fb.com/publications/canopy-end-to-end-
>>> performance-tracing-at-scale/)
>>>
>>> As a bonus, they referenced Accumulo and HTrace in section 2.2
>>>
>>> "Mismatched models affected compatibility between mixed system versions;
>>> e.g. Accumulo and Hadoop were impacted by the “continued lack of concern
>>> in
>>> the HTrace project around tracing during upgrades”
>>>
>>>
>>> -----Original Message-----
>>> From: Tony Kurc [mailto:tkurc@apache.org]
>>> Sent: Tuesday, February 27, 2018 12:57 PM
>>> To: dev@accumulo.apache.org
>>> Subject: Re: [DISCUSS] tracing framework updates
>>>
>>> I have some experience with opentracing, and it definitely seems
>>> promising, however, potentially promising in the same way htrace was...
>>> That being said, I did a cursory thought exercise of what it would take
>>> to
>>> do a swap of the current tracing in accumulo to opentracing, and I didn't
>>> come across any hard problems, meaning it could be a fairly
>>> straightforward
>>> refactor. I was hoping to explore the community a bit more at some
>>> upcoming
>>> conferences
>>>
>>> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>>>
>>>
>>>
>>>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>>>
>>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>>>> us
>>>>>
>>>>> can
>>>>
>>>> start participating? The project did start within Accumulo, after all.
>>>>>
>>>>> What
>>>>
>>>> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>>>>>
>>>>>
>>>>> I suspect it's too late for HTrace. The last commit to the main
>>>> development branch was May 2017. They had a decent run of activity in
>>>> 2015 and an almost-resurgence in 2016, but they never really got
>>>> enough community traction to survive the normal ebb and flow of
>>>> contributor involvement.
>>>>
>>>> They need the things any project needs to be sustainable: regular
>>>> release cadences, a responsive contribution process, and folks to do
>>>> the long slog of building interest via e.g. production adoption.
>>>>
>>>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>>>
>>>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>>>> HTrace is
>>>>>
>>>>> actually
>>>>
>>>> listed as an instrumentation library for Zipkin (among others).
>>>>>
>>>>>
>>>>> I think the key is that for a instrumentation library to get adoption
>>>> it needs a good sink that provides utility to operators looking to
>>>> diagnose problems. It took too long for HTrace to provide any tooling
>>>> that could help with even simple performance profiling. Maybe hooking
>>>> it into Zipkin would get around that. Personally, I never managed to
>>>> get the two to actually work together.
>>>>
>>>> My listing Zipkin as an option merely reflects my prioritization of
>>>> practical impact of whatever we go to. I don't want to adopt some
>>>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>>>> a zipkin-sink compatible runtime.
>>>>
>>>> There's a whole community that just does distributed monitoring, maybe
>>>> someone has time to survey some spaces and see if OpenTracing has any
>>>> legs.
>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSS] tracing framework updates
Posted by Josh Elser <el...@apache.org>.
Oh, that's a pleasant surprise to hear, actually.
Anything you can share with the class, Tony? Would love to hear (even if
brief) how it was used and benefited you.
Specifically, I'm curious if...
* You looked at traces from our server-side instrumented code
* You instrumented your own code outside of Accumulo and used Accumulo
as the backing store
* You instrumented code inside/outside Accumulo and benefited from the
server-side instrumentation (e.g. your code's spans collapsing with the
server's spans)
On 2/27/18 6:52 PM, Tony Kurc wrote:
> I'd personally be disappointed to see it removed. There is a bit of a
> learning curve and startup cost to use it now, but when diagnosing major
> challenges, it has been an invaluable capability.
>
> On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
>
> Wow... that's, erm, quite the paper. Nothing like taking some pot-shots at
> another software project and quoting folks out of context.
>
> Does it help to break down the problem some more?
>
> * Is Accumulo getting benefit from tracing its library?
> * Is Accumulo getting benefit from tracing context including HDFS calls?
>
> I feel like it is a nice tool to have in your toolbelt (having used it
> successfully in the past), but I wonder if it's the most effective thing to
> keep inside of Accumulo. Specifically, would it be better to just pull this
> out of Accumulo outright?
>
> I don't think I have an opinion yet.
>
>
> On 2/27/18 1:08 PM, Ed Coleman wrote:
>
>> For general discussion - Facebook recently (Oct 28, 2017) published a
>> paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
>> System (https://research.fb.com/publications/canopy-end-to-end-
>> performance-tracing-at-scale/)
>>
>> As a bonus, they referenced Accumulo and HTrace in section 2.2
>>
>> "Mismatched models affected compatibility between mixed system versions;
>> e.g. Accumulo and Hadoop were impacted by the “continued lack of concern in
>> the HTrace project around tracing during upgrades”
>>
>>
>> -----Original Message-----
>> From: Tony Kurc [mailto:tkurc@apache.org]
>> Sent: Tuesday, February 27, 2018 12:57 PM
>> To: dev@accumulo.apache.org
>> Subject: Re: [DISCUSS] tracing framework updates
>>
>> I have some experience with opentracing, and it definitely seems
>> promising, however, potentially promising in the same way htrace was...
>> That being said, I did a cursory thought exercise of what it would take to
>> do a swap of the current tracing in accumulo to opentracing, and I didn't
>> come across any hard problems, meaning it could be a fairly straightforward
>> refactor. I was hoping to explore the community a bit more at some upcoming
>> conferences
>>
>> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>>
>>
>>>
>>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>>
>>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>>> us
>>>>
>>> can
>>>
>>>> start participating? The project did start within Accumulo, after all.
>>>>
>>> What
>>>
>>>> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>>>>
>>>>
>>> I suspect it's too late for HTrace. The last commit to the main
>>> development branch was May 2017. They had a decent run of activity in
>>> 2015 and an almost-resurgence in 2016, but they never really got
>>> enough community traction to survive the normal ebb and flow of
>>> contributor involvement.
>>>
>>> They need the things any project needs to be sustainable: regular
>>> release cadences, a responsive contribution process, and folks to do
>>> the long slog of building interest via e.g. production adoption.
>>>
>>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>>> HTrace is
>>>>
>>> actually
>>>
>>>> listed as an instrumentation library for Zipkin (among others).
>>>>
>>>>
>>> I think the key is that for a instrumentation library to get adoption
>>> it needs a good sink that provides utility to operators looking to
>>> diagnose problems. It took too long for HTrace to provide any tooling
>>> that could help with even simple performance profiling. Maybe hooking
>>> it into Zipkin would get around that. Personally, I never managed to
>>> get the two to actually work together.
>>>
>>> My listing Zipkin as an option merely reflects my prioritization of
>>> practical impact of whatever we go to. I don't want to adopt some
>>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>>> a zipkin-sink compatible runtime.
>>>
>>> There's a whole community that just does distributed monitoring, maybe
>>> someone has time to survey some spaces and see if OpenTracing has any
>>> legs.
>>>
>>>
>>
>
Re: [DISCUSS] tracing framework updates
Posted by Tony Kurc <tr...@gmail.com>.
I'd personally be disappointed to see it removed. There is a bit of a
learning curve and startup cost to use it now, but when diagnosing major
challenges, it has been an invaluable capability.
On Feb 27, 2018 3:15 PM, "Josh Elser" <el...@apache.org> wrote:
Wow... that's, erm, quite the paper. Nothing like taking some pot-shots at
another software project and quoting folks out of context.
Does it help to break down the problem some more?
* Is Accumulo getting benefit from tracing its library?
* Is Accumulo getting benefit from tracing context including HDFS calls?
I feel like it is a nice tool to have in your toolbelt (having used it
successfully in the past), but I wonder if it's the most effective thing to
keep inside of Accumulo. Specifically, would it be better to just pull this
out of Accumulo outright?
I don't think I have an opinion yet.
On 2/27/18 1:08 PM, Ed Coleman wrote:
> For general discussion - Facebook recently (Oct 28, 2017) published a
> paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis
> System (https://research.fb.com/publications/canopy-end-to-end-
> performance-tracing-at-scale/)
>
> As a bonus, they referenced Accumulo and HTrace in section 2.2
>
> "Mismatched models affected compatibility between mixed system versions;
> e.g. Accumulo and Hadoop were impacted by the “continued lack of concern in
> the HTrace project around tracing during upgrades”
>
>
> -----Original Message-----
> From: Tony Kurc [mailto:tkurc@apache.org]
> Sent: Tuesday, February 27, 2018 12:57 PM
> To: dev@accumulo.apache.org
> Subject: Re: [DISCUSS] tracing framework updates
>
> I have some experience with opentracing, and it definitely seems
> promising, however, potentially promising in the same way htrace was...
> That being said, I did a cursory thought exercise of what it would take to
> do a swap of the current tracing in accumulo to opentracing, and I didn't
> come across any hard problems, meaning it could be a fairly straightforward
> refactor. I was hoping to explore the community a bit more at some upcoming
> conferences
>
> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>
>
>>
>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>
>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>> us
>>>
>> can
>>
>>> start participating? The project did start within Accumulo, after all.
>>>
>> What
>>
>>> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>>>
>>>
>> I suspect it's too late for HTrace. The last commit to the main
>> development branch was May 2017. They had a decent run of activity in
>> 2015 and an almost-resurgence in 2016, but they never really got
>> enough community traction to survive the normal ebb and flow of
>> contributor involvement.
>>
>> They need the things any project needs to be sustainable: regular
>> release cadences, a responsive contribution process, and folks to do
>> the long slog of building interest via e.g. production adoption.
>>
>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>> HTrace is
>>>
>> actually
>>
>>> listed as an instrumentation library for Zipkin (among others).
>>>
>>>
>> I think the key is that for a instrumentation library to get adoption
>> it needs a good sink that provides utility to operators looking to
>> diagnose problems. It took too long for HTrace to provide any tooling
>> that could help with even simple performance profiling. Maybe hooking
>> it into Zipkin would get around that. Personally, I never managed to
>> get the two to actually work together.
>>
>> My listing Zipkin as an option merely reflects my prioritization of
>> practical impact of whatever we go to. I don't want to adopt some
>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>> a zipkin-sink compatible runtime.
>>
>> There's a whole community that just does distributed monitoring, maybe
>> someone has time to survey some spaces and see if OpenTracing has any
>> legs.
>>
>>
>
Re: [DISCUSS] tracing framework updates
Posted by Josh Elser <el...@apache.org>.
Wow... that's, erm, quite the paper. Nothing like taking some pot-shots
at another software project and quoting folks out of context.
Does it help to break down the problem some more?
* Is Accumulo getting benefit from tracing its library?
* Is Accumulo getting benefit from tracing context including HDFS calls?
I feel like it is a nice tool to have in your toolbelt (having used it
successfully in the past), but I wonder if it's the most effective thing
to keep inside of Accumulo. Specifically, would it be better to just
pull this out of Accumulo outright?
I don't think I have an opinion yet.
On 2/27/18 1:08 PM, Ed Coleman wrote:
> For general discussion - Facebook recently (Oct 28, 2017) published a paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis System (https://research.fb.com/publications/canopy-end-to-end-performance-tracing-at-scale/)
>
> As a bonus, they referenced Accumulo and HTrace in section 2.2
>
> "Mismatched models affected compatibility between mixed system versions; e.g. Accumulo and Hadoop were impacted by the “continued lack of concern in the HTrace project around tracing during upgrades”
>
>
> -----Original Message-----
> From: Tony Kurc [mailto:tkurc@apache.org]
> Sent: Tuesday, February 27, 2018 12:57 PM
> To: dev@accumulo.apache.org
> Subject: Re: [DISCUSS] tracing framework updates
>
> I have some experience with opentracing, and it definitely seems promising, however, potentially promising in the same way htrace was... That being said, I did a cursory thought exercise of what it would take to do a swap of the current tracing in accumulo to opentracing, and I didn't come across any hard problems, meaning it could be a fairly straightforward refactor. I was hoping to explore the community a bit more at some upcoming conferences
>
> On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>
>>
>>
>> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
>>> I didn't realize HTrace was struggling in incubation. Maybe some of
>>> us
>> can
>>> start participating? The project did start within Accumulo, after all.
>> What
>>> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>>>
>>
>> I suspect it's too late for HTrace. The last commit to the main
>> development branch was May 2017. They had a decent run of activity in
>> 2015 and an almost-resurgence in 2016, but they never really got
>> enough community traction to survive the normal ebb and flow of
>> contributor involvement.
>>
>> They need the things any project needs to be sustainable: regular
>> release cadences, a responsive contribution process, and folks to do
>> the long slog of building interest via e.g. production adoption.
>>
>>> I'm unfamiliar with OpenTracing, but it was my understanding that
>>> Zipkin was more of a tracing sink, than an instrumentation API.
>>> HTrace is
>> actually
>>> listed as an instrumentation library for Zipkin (among others).
>>>
>>
>> I think the key is that for a instrumentation library to get adoption
>> it needs a good sink that provides utility to operators looking to
>> diagnose problems. It took too long for HTrace to provide any tooling
>> that could help with even simple performance profiling. Maybe hooking
>> it into Zipkin would get around that. Personally, I never managed to
>> get the two to actually work together.
>>
>> My listing Zipkin as an option merely reflects my prioritization of
>> practical impact of whatever we go to. I don't want to adopt some
>> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
>> a zipkin-sink compatible runtime.
>>
>> There's a whole community that just does distributed monitoring, maybe
>> someone has time to survey some spaces and see if OpenTracing has any legs.
>>
>
RE: [DISCUSS] tracing framework updates
Posted by Ed Coleman <de...@etcoleman.com>.
For general discussion - Facebook recently (Oct 28, 2017) published a paper on tracing: Canopy: An End-to-End Performance Tracing and Analysis System (https://research.fb.com/publications/canopy-end-to-end-performance-tracing-at-scale/)
As a bonus, they referenced Accumulo and HTrace in section 2.2
"Mismatched models affected compatibility between mixed system versions; e.g. Accumulo and Hadoop were impacted by the “continued lack of concern in the HTrace project around tracing during upgrades”
-----Original Message-----
From: Tony Kurc [mailto:tkurc@apache.org]
Sent: Tuesday, February 27, 2018 12:57 PM
To: dev@accumulo.apache.org
Subject: Re: [DISCUSS] tracing framework updates
I have some experience with opentracing, and it definitely seems promising, however, potentially promising in the same way htrace was... That being said, I did a cursory thought exercise of what it would take to do a swap of the current tracing in accumulo to opentracing, and I didn't come across any hard problems, meaning it could be a fairly straightforward refactor. I was hoping to explore the community a bit more at some upcoming conferences
On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>
>
> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
> > I didn't realize HTrace was struggling in incubation. Maybe some of
> > us
> can
> > start participating? The project did start within Accumulo, after all.
> What
> > does it need? I also wouldn't want to go back to maintaining cloudtrace.
> >
>
> I suspect it's too late for HTrace. The last commit to the main
> development branch was May 2017. They had a decent run of activity in
> 2015 and an almost-resurgence in 2016, but they never really got
> enough community traction to survive the normal ebb and flow of
> contributor involvement.
>
> They need the things any project needs to be sustainable: regular
> release cadences, a responsive contribution process, and folks to do
> the long slog of building interest via e.g. production adoption.
>
> > I'm unfamiliar with OpenTracing, but it was my understanding that
> > Zipkin was more of a tracing sink, than an instrumentation API.
> > HTrace is
> actually
> > listed as an instrumentation library for Zipkin (among others).
> >
>
> I think the key is that for a instrumentation library to get adoption
> it needs a good sink that provides utility to operators looking to
> diagnose problems. It took too long for HTrace to provide any tooling
> that could help with even simple performance profiling. Maybe hooking
> it into Zipkin would get around that. Personally, I never managed to
> get the two to actually work together.
>
> My listing Zipkin as an option merely reflects my prioritization of
> practical impact of whatever we go to. I don't want to adopt some
> blue-sky effort. FWIW, OpenTracing docs at least claim to also provide
> a zipkin-sink compatible runtime.
>
> There's a whole community that just does distributed monitoring, maybe
> someone has time to survey some spaces and see if OpenTracing has any legs.
>
Re: [DISCUSS] tracing framework updates
Posted by Tony Kurc <tk...@apache.org>.
I have some experience with opentracing, and it definitely seems promising,
however, potentially promising in the same way htrace was... That being
said, I did a cursory thought exercise of what it would take to do a swap
of the current tracing in accumulo to opentracing, and I didn't come across
any hard problems, meaning it could be a fairly straightforward refactor. I
was hoping to explore the community a bit more at some upcoming conferences
On Feb 27, 2018 11:59 AM, "Sean Busbey" <bu...@apache.org> wrote:
>
>
> On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
> > I didn't realize HTrace was struggling in incubation. Maybe some of us
> can
> > start participating? The project did start within Accumulo, after all.
> What
> > does it need? I also wouldn't want to go back to maintaining cloudtrace.
> >
>
> I suspect it's too late for HTrace. The last commit to the main
> development branch was May 2017. They had a decent run of activity in 2015
> and an almost-resurgence in 2016, but they never really got enough
> community traction to survive the normal ebb and flow of contributor
> involvement.
>
> They need the things any project needs to be sustainable: regular release
> cadences, a responsive contribution process, and folks to do the long slog
> of building interest via e.g. production adoption.
>
> > I'm unfamiliar with OpenTracing, but it was my understanding that Zipkin
> > was more of a tracing sink, than an instrumentation API. HTrace is
> actually
> > listed as an instrumentation library for Zipkin (among others).
> >
>
> I think the key is that for a instrumentation library to get adoption it
> needs a good sink that provides utility to operators looking to diagnose
> problems. It took too long for HTrace to provide any tooling that could
> help with even simple performance profiling. Maybe hooking it into Zipkin
> would get around that. Personally, I never managed to get the two to
> actually work together.
>
> My listing Zipkin as an option merely reflects my prioritization of
> practical impact of whatever we go to. I don't want to adopt some blue-sky
> effort. FWIW, OpenTracing docs at least claim to also provide a zipkin-sink
> compatible runtime.
>
> There's a whole community that just does distributed monitoring, maybe
> someone has time to survey some spaces and see if OpenTracing has any legs.
>
Re: [DISCUSS] tracing framework updates
Posted by Sean Busbey <bu...@apache.org>.
On 2018/02/27 16:39:02, Christopher <ct...@apache.org> wrote:
> I didn't realize HTrace was struggling in incubation. Maybe some of us can
> start participating? The project did start within Accumulo, after all. What
> does it need? I also wouldn't want to go back to maintaining cloudtrace.
>
I suspect it's too late for HTrace. The last commit to the main development branch was May 2017. They had a decent run of activity in 2015 and an almost-resurgence in 2016, but they never really got enough community traction to survive the normal ebb and flow of contributor involvement.
They need the things any project needs to be sustainable: regular release cadences, a responsive contribution process, and folks to do the long slog of building interest via e.g. production adoption.
> I'm unfamiliar with OpenTracing, but it was my understanding that Zipkin
> was more of a tracing sink, than an instrumentation API. HTrace is actually
> listed as an instrumentation library for Zipkin (among others).
>
I think the key is that for a instrumentation library to get adoption it needs a good sink that provides utility to operators looking to diagnose problems. It took too long for HTrace to provide any tooling that could help with even simple performance profiling. Maybe hooking it into Zipkin would get around that. Personally, I never managed to get the two to actually work together.
My listing Zipkin as an option merely reflects my prioritization of practical impact of whatever we go to. I don't want to adopt some blue-sky effort. FWIW, OpenTracing docs at least claim to also provide a zipkin-sink compatible runtime.
There's a whole community that just does distributed monitoring, maybe someone has time to survey some spaces and see if OpenTracing has any legs.
Re: [DISCUSS] tracing framework updates
Posted by Christopher <ct...@apache.org>.
I didn't realize HTrace was struggling in incubation. Maybe some of us can
start participating? The project did start within Accumulo, after all. What
does it need? I also wouldn't want to go back to maintaining cloudtrace.
I'm unfamiliar with OpenTracing, but it was my understanding that Zipkin
was more of a tracing sink, than an instrumentation API. HTrace is actually
listed as an instrumentation library for Zipkin (among others).
On Tue, Feb 27, 2018 at 11:23 AM Sean Busbey <bu...@apache.org> wrote:
> More things we should get ahead of for Accumulo 2.0.0: distributed tracing.
>
> Right now we have an awkward situation wrt HTrace support. We're using and
> shipping htrace 3.1. It works okay for our internal uses, afaict.
>
> Hadoop 2.6 ships and uses HTrace 3.0. I believe this does not work with
> 3.1.
>
> Hadoop 2.7 ships and uses HTrace 3.1, which means we can trace things down
> through the hdfs layer, though I've never managed to get anything useful
> out of trying to do so.
>
> Hadoop 2.8+ ships and uses HTrace 4.0, which isn't compatible with 3.1.
>
> Hadoop 2.9+ and 3.0+ ship and use HTrace 4.1, it doesn't work with 3.1 and
> it's not clear to me if it works with 4.0.
>
> I *think* this means that we can only do tracing down into a single minor
> release line of HDFS.
>
> Additionally, as of their January podling report[1] it looks like HTrace
> is about ready to shut down as a project. In theory it could continue on
> github or something after exiting the ASF incubator, but I don't see why
> we'd expect an increase in activity; the project's been pretty idle for at
> least a year.
>
> I really don't want to go back to maintaining cloudtrace. So what should
> we do? Zipkin has some decent uptake[2]; they're still making regular
> releases. Maybe just adopt OpenTracing[3] as a way to hedge against having
> to do his again?
>
> YCSB is the Accumulo downstream with tracing that I'm most familiar with.
> It relies on HTrace 4.1. I'm probably going to push that community to
> abandon HTrace soon. (I was the one that pushed it to adopt HTrace in the
> first place). I'd like to at least get tracing from YCSB through Accumulo
> in the process.
>
> [1]: https://s.apache.org/uDHN
> [2]: https://zipkin.io/
> [3]: http://opentracing.io/documentation/
>