You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by stalary <st...@163.com> on 2022/05/15 11:13:49 UTC
[Discuss][DSIP] Improve Doris observability
Dear All,
I’d like to improve Doris observability. This is achieved by introducing Skywalking and OpenTelemetry. Parameters allow users to decide which monitoring tool to use. First I want to support trace on the FE, implement tracing of SQL statements.
The associated issue：https://github.com/apache/incubator-doris/issues/9577
Here are the results of my preliminary research
1. Currently OpenTelemetry is very convincing in terms of protocol specifications, including traces, metrics, logs and so on, But it does not provide native data visualization, we need to export data to Zipkin or Prometheus to visualize.
2. Skywalking is currently popular, active, and has excellent feature support, including data visualization, traces, metrics, logs and so on, in particular, there is excellent support for data visualization
3. Both have Java and c++ sdk that Doris can use
—
Best regards
Stalary
Re:Re:Re:Re: Re: [Discuss][DSIP] Improve Doris observability
Posted by 陈明雨 <mo...@163.com>.
Hi Zenglin,
I have create a DSIP for this proposal[1].
[1] https://cwiki.apache.org/confluence/display/DORIS/DSIP-012%3A+Introduce+opentelemetry
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
在 2022-05-17 18:54:21，"luozenglin" <18...@163.com> 写道：
>Hi Mingyu,
>I have registered the wiki, this is my userid: Zenglin Luo.
>thank you.
>
>
>by luozenglin
>
>At 2022-05-17 12:17:29, "陈明雨" <mo...@163.com> wrote:
>>Hi Zenglin,
>>I would like to create a DSIP for this feature.
>>Could you please register the wiki: https://cwiki.apache.org/confluence/display/DORIS/Doris+Improvement+Proposals
>>and give me your userid? I will open the write permission to you
>>
>>
>>
>>
>>--
>>
>>此致！Best Regards
>>陈明雨 Mingyu Chen
>>
>>Email:
>>chenmingyu@apache.org
>>
>>
>>
>>
>>
>>在 2022-05-16 20:09:11，"Willem Jiang" <wi...@gmail.com> 写道：
>>>Could you write a proposal with more detail information of implementation？
>>>So we can review it before you starting the code writing？
>>>
>>>Willem Jiang
>>>
>>>Twitter: willemjiang
>>>Weibo: 姜宁willem
>>>
>>>On Mon, May 16, 2022 at 8:02 PM luozenglin <18...@163.com> wrote:
>>>>
>>>> Hi Stalary,
>>>> The idea of introducing OpenTelemetry to improve Doris observability sounds cool. Actually, I'm already doing related development work. I have built a code framework to collect and send query trace data, it still has a lot of refinement work to do. I'll commit the code to to github for discussion later. You are very welcome to participate in the work of doris observability and look forward to developing with you.
>>>>
>>>>
>>>>
>>>>
>>>> At 2022-05-16 17:58:19, "stalary" <st...@163.com> wrote:
>>>> >Thank you very much, I will communicate with Luozenglin.
>>>> >
>>>> >
>>>> >by stalary
>>>> >
>>>> >
>>>> >On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
>>>> >Hi stalary
>>>> >luozenglin <https://github.com/luozenglin> has done a lot of work on
>>>> >OpenTelemetry and trace before, and has made some progress, I think you can
>>>> >work with him to develop this thing together
>>>> >Thanks
>>>> >Yang Zhengguo
>>>> >
>>>> >
>>>> >stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
>>>> >
>>>> >I will split it for five steps, step1 will be visualized first.
>>>> >Step1. Sql Plan
>>>> >Step2. Execute Profile and data integration
>>>> >Step3. Doris topology
>>>> >Step4. Doris fe and be metrics
>>>> >Step5. Doris log
>>>> >
>>>> >
>>>> >by stalary
>>>> >
>>>> >
>>>> >On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
>>>> >Hi Stalary,
>>>> >What kind of data will be visualized first?
>>>> >Could you integrate profile data and sql plan together？
>>>> >Any way, this is a good idea!
>>>> >
>>>> >
>>>> >Minghong
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
>>>> >nice work！
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >------------------ Original ------------------
>>>> >From: stalary <stalary@163.com>
>>>> >Date: Sun,May 15,2022 7:13 PM
>>>> >To: dev@doris.apache.org <dev@doris.apache.org>
>>>> >Subject: Re: [Discuss][DSIP] Improve Doris observability
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >Dear All,
>>>> >
>>>> >I’d like to improve Doris observability. This is achieved by introducing
>>>> >Skywalking and OpenTelemetry. Parameters allow users to decide which
>>>> >monitoring tool to use. First I want to support trace on the FE, implement
>>>> >tracing of SQL statements.
>>>> >
>>>> >The associated issue：https://github.com/apache/incubator-doris/issues/9577
>>>> >
>>>> >Here are the results of my preliminary research
>>>> >1. Currently OpenTelemetry is very convincing in terms of protocol
>>>> >specifications, including traces, metrics, logs and so on, But it does not
>>>> >provide native data visualization, we need to export data to Zipkin or
>>>> >Prometheus to visualize.
>>>> >
>>>> >
>>>> >2. Skywalking is currently popular, active, and has excellent feature
>>>> >support, including data visualization, traces, metrics, logs and so on, in
>>>> >particular, there is excellent support for data visualization
>>>> >
>>>> >
>>>> >3. Both have Java and c++ sdk that Doris can use
>>>> >
>>>> >—
>>>> >Best regards
>>>> >Stalary
>>>> >
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
>>>For additional commands, e-mail: dev-help@doris.apache.org
Re:Re:Re: Re: [Discuss][DSIP] Improve Doris observability
Posted by luozenglin <18...@163.com>.
Hi Mingyu,
I have registered the wiki, this is my userid: Zenglin Luo.
thank you.
by luozenglin
At 2022-05-17 12:17:29, "陈明雨" <mo...@163.com> wrote:
>Hi Zenglin,
>I would like to create a DSIP for this feature.
>Could you please register the wiki: https://cwiki.apache.org/confluence/display/DORIS/Doris+Improvement+Proposals
>and give me your userid? I will open the write permission to you
>
>
>
>
>--
>
>此致！Best Regards
>陈明雨 Mingyu Chen
>
>Email:
>chenmingyu@apache.org
>
>
>
>
>
>在 2022-05-16 20:09:11，"Willem Jiang" <wi...@gmail.com> 写道：
>>Could you write a proposal with more detail information of implementation？
>>So we can review it before you starting the code writing？
>>
>>Willem Jiang
>>
>>Twitter: willemjiang
>>Weibo: 姜宁willem
>>
>>On Mon, May 16, 2022 at 8:02 PM luozenglin <18...@163.com> wrote:
>>>
>>> Hi Stalary,
>>> The idea of introducing OpenTelemetry to improve Doris observability sounds cool. Actually, I'm already doing related development work. I have built a code framework to collect and send query trace data, it still has a lot of refinement work to do. I'll commit the code to to github for discussion later. You are very welcome to participate in the work of doris observability and look forward to developing with you.
>>>
>>>
>>>
>>>
>>> At 2022-05-16 17:58:19, "stalary" <st...@163.com> wrote:
>>> >Thank you very much, I will communicate with Luozenglin.
>>> >
>>> >
>>> >by stalary
>>> >
>>> >
>>> >On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
>>> >Hi stalary
>>> >luozenglin <https://github.com/luozenglin> has done a lot of work on
>>> >OpenTelemetry and trace before, and has made some progress, I think you can
>>> >work with him to develop this thing together
>>> >Thanks
>>> >Yang Zhengguo
>>> >
>>> >
>>> >stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
>>> >
>>> >I will split it for five steps, step1 will be visualized first.
>>> >Step1. Sql Plan
>>> >Step2. Execute Profile and data integration
>>> >Step3. Doris topology
>>> >Step4. Doris fe and be metrics
>>> >Step5. Doris log
>>> >
>>> >
>>> >by stalary
>>> >
>>> >
>>> >On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
>>> >Hi Stalary,
>>> >What kind of data will be visualized first?
>>> >Could you integrate profile data and sql plan together？
>>> >Any way, this is a good idea!
>>> >
>>> >
>>> >Minghong
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
>>> >nice work！
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >------------------ Original ------------------
>>> >From: stalary <stalary@163.com>
>>> >Date: Sun,May 15,2022 7:13 PM
>>> >To: dev@doris.apache.org <dev@doris.apache.org>
>>> >Subject: Re: [Discuss][DSIP] Improve Doris observability
>>> >
>>> >
>>> >
>>> >
>>> >Dear All,
>>> >
>>> >I’d like to improve Doris observability. This is achieved by introducing
>>> >Skywalking and OpenTelemetry. Parameters allow users to decide which
>>> >monitoring tool to use. First I want to support trace on the FE, implement
>>> >tracing of SQL statements.
>>> >
>>> >The associated issue：https://github.com/apache/incubator-doris/issues/9577
>>> >
>>> >Here are the results of my preliminary research
>>> >1. Currently OpenTelemetry is very convincing in terms of protocol
>>> >specifications, including traces, metrics, logs and so on, But it does not
>>> >provide native data visualization, we need to export data to Zipkin or
>>> >Prometheus to visualize.
>>> >
>>> >
>>> >2. Skywalking is currently popular, active, and has excellent feature
>>> >support, including data visualization, traces, metrics, logs and so on, in
>>> >particular, there is excellent support for data visualization
>>> >
>>> >
>>> >3. Both have Java and c++ sdk that Doris can use
>>> >
>>> >—
>>> >Best regards
>>> >Stalary
>>> >
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
>>For additional commands, e-mail: dev-help@doris.apache.org
Re:Re: Re: [Discuss][DSIP] Improve Doris observability
Posted by 陈明雨 <mo...@163.com>.
Hi Zenglin,
I would like to create a DSIP for this feature.
Could you please register the wiki: https://cwiki.apache.org/confluence/display/DORIS/Doris+Improvement+Proposals
and give me your userid? I will open the write permission to you
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
在 2022-05-16 20:09:11，"Willem Jiang" <wi...@gmail.com> 写道：
>Could you write a proposal with more detail information of implementation？
>So we can review it before you starting the code writing？
>
>Willem Jiang
>
>Twitter: willemjiang
>Weibo: 姜宁willem
>
>On Mon, May 16, 2022 at 8:02 PM luozenglin <18...@163.com> wrote:
>>
>> Hi Stalary,
>> The idea of introducing OpenTelemetry to improve Doris observability sounds cool. Actually, I'm already doing related development work. I have built a code framework to collect and send query trace data, it still has a lot of refinement work to do. I'll commit the code to to github for discussion later. You are very welcome to participate in the work of doris observability and look forward to developing with you.
>>
>>
>>
>>
>> At 2022-05-16 17:58:19, "stalary" <st...@163.com> wrote:
>> >Thank you very much, I will communicate with Luozenglin.
>> >
>> >
>> >by stalary
>> >
>> >
>> >On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
>> >Hi stalary
>> >luozenglin <https://github.com/luozenglin> has done a lot of work on
>> >OpenTelemetry and trace before, and has made some progress, I think you can
>> >work with him to develop this thing together
>> >Thanks
>> >Yang Zhengguo
>> >
>> >
>> >stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
>> >
>> >I will split it for five steps, step1 will be visualized first.
>> >Step1. Sql Plan
>> >Step2. Execute Profile and data integration
>> >Step3. Doris topology
>> >Step4. Doris fe and be metrics
>> >Step5. Doris log
>> >
>> >
>> >by stalary
>> >
>> >
>> >On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
>> >Hi Stalary,
>> >What kind of data will be visualized first?
>> >Could you integrate profile data and sql plan together？
>> >Any way, this is a good idea!
>> >
>> >
>> >Minghong
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
>> >nice work！
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >------------------ Original ------------------
>> >From: stalary <stalary@163.com>
>> >Date: Sun,May 15,2022 7:13 PM
>> >To: dev@doris.apache.org <dev@doris.apache.org>
>> >Subject: Re: [Discuss][DSIP] Improve Doris observability
>> >
>> >
>> >
>> >
>> >Dear All,
>> >
>> >I’d like to improve Doris observability. This is achieved by introducing
>> >Skywalking and OpenTelemetry. Parameters allow users to decide which
>> >monitoring tool to use. First I want to support trace on the FE, implement
>> >tracing of SQL statements.
>> >
>> >The associated issue：https://github.com/apache/incubator-doris/issues/9577
>> >
>> >Here are the results of my preliminary research
>> >1. Currently OpenTelemetry is very convincing in terms of protocol
>> >specifications, including traces, metrics, logs and so on, But it does not
>> >provide native data visualization, we need to export data to Zipkin or
>> >Prometheus to visualize.
>> >
>> >
>> >2. Skywalking is currently popular, active, and has excellent feature
>> >support, including data visualization, traces, metrics, logs and so on, in
>> >particular, there is excellent support for data visualization
>> >
>> >
>> >3. Both have Java and c++ sdk that Doris can use
>> >
>> >—
>> >Best regards
>> >Stalary
>> >
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
>For additional commands, e-mail: dev-help@doris.apache.org
Re: Re: [Discuss][DSIP] Improve Doris observability
Posted by Willem Jiang <wi...@gmail.com>.
Could you write a proposal with more detail information of implementation？
So we can review it before you starting the code writing？
Willem Jiang
Twitter: willemjiang
Weibo: 姜宁willem
On Mon, May 16, 2022 at 8:02 PM luozenglin <18...@163.com> wrote:
>
> Hi Stalary,
> The idea of introducing OpenTelemetry to improve Doris observability sounds cool. Actually, I'm already doing related development work. I have built a code framework to collect and send query trace data, it still has a lot of refinement work to do. I'll commit the code to to github for discussion later. You are very welcome to participate in the work of doris observability and look forward to developing with you.
>
>
>
>
> At 2022-05-16 17:58:19, "stalary" <st...@163.com> wrote:
> >Thank you very much, I will communicate with Luozenglin.
> >
> >
> >by stalary
> >
> >
> >On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
> >Hi stalary
> >luozenglin <https://github.com/luozenglin> has done a lot of work on
> >OpenTelemetry and trace before, and has made some progress, I think you can
> >work with him to develop this thing together
> >Thanks
> >Yang Zhengguo
> >
> >
> >stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
> >
> >I will split it for five steps, step1 will be visualized first.
> >Step1. Sql Plan
> >Step2. Execute Profile and data integration
> >Step3. Doris topology
> >Step4. Doris fe and be metrics
> >Step5. Doris log
> >
> >
> >by stalary
> >
> >
> >On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
> >Hi Stalary,
> >What kind of data will be visualized first?
> >Could you integrate profile data and sql plan together？
> >Any way, this is a good idea!
> >
> >
> >Minghong
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
> >nice work！
> >
> >
> >
> >
> >
> >
> >
> >------------------ Original ------------------
> >From: stalary <stalary@163.com>
> >Date: Sun,May 15,2022 7:13 PM
> >To: dev@doris.apache.org <dev@doris.apache.org>
> >Subject: Re: [Discuss][DSIP] Improve Doris observability
> >
> >
> >
> >
> >Dear All,
> >
> >I’d like to improve Doris observability. This is achieved by introducing
> >Skywalking and OpenTelemetry. Parameters allow users to decide which
> >monitoring tool to use. First I want to support trace on the FE, implement
> >tracing of SQL statements.
> >
> >The associated issue：https://github.com/apache/incubator-doris/issues/9577
> >
> >Here are the results of my preliminary research
> >1. Currently OpenTelemetry is very convincing in terms of protocol
> >specifications, including traces, metrics, logs and so on, But it does not
> >provide native data visualization, we need to export data to Zipkin or
> >Prometheus to visualize.
> >
> >
> >2. Skywalking is currently popular, active, and has excellent feature
> >support, including data visualization, traces, metrics, logs and so on, in
> >particular, there is excellent support for data visualization
> >
> >
> >3. Both have Java and c++ sdk that Doris can use
> >
> >—
> >Best regards
> >Stalary
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re:Re: [Discuss][DSIP] Improve Doris observability
Posted by luozenglin <18...@163.com>.
Hi Stalary,
The idea of introducing OpenTelemetry to improve Doris observability sounds cool. Actually, I'm already doing related development work. I have built a code framework to collect and send query trace data, it still has a lot of refinement work to do. I'll commit the code to to github for discussion later. You are very welcome to participate in the work of doris observability and look forward to developing with you.
At 2022-05-16 17:58:19, "stalary" <st...@163.com> wrote:
>Thank you very much, I will communicate with Luozenglin.
>
>
>by stalary
>
>
>On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
>Hi stalary
>luozenglin <https://github.com/luozenglin> has done a lot of work on
>OpenTelemetry and trace before, and has made some progress, I think you can
>work with him to develop this thing together
>Thanks
>Yang Zhengguo
>
>
>stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
>
>I will split it for five steps, step1 will be visualized first.
>Step1. Sql Plan
>Step2. Execute Profile and data integration
>Step3. Doris topology
>Step4. Doris fe and be metrics
>Step5. Doris log
>
>
>by stalary
>
>
>On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
>Hi Stalary,
>What kind of data will be visualized first?
>Could you integrate profile data and sql plan together？
>Any way, this is a good idea!
>
>
>Minghong
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
>nice work！
>
>
>
>
>
>
>
>------------------ Original ------------------
>From: stalary <stalary@163.com>
>Date: Sun,May 15,2022 7:13 PM
>To: dev@doris.apache.org <dev@doris.apache.org>
>Subject: Re: [Discuss][DSIP] Improve Doris observability
>
>
>
>
>Dear All,
>
>I’d like to improve Doris observability. This is achieved by introducing
>Skywalking and OpenTelemetry. Parameters allow users to decide which
>monitoring tool to use. First I want to support trace on the FE, implement
>tracing of SQL statements.
>
>The associated issue：https://github.com/apache/incubator-doris/issues/9577
>
>Here are the results of my preliminary research
>1. Currently OpenTelemetry is very convincing in terms of protocol
>specifications, including traces, metrics, logs and so on, But it does not
>provide native data visualization, we need to export data to Zipkin or
>Prometheus to visualize.
>
>
>2. Skywalking is currently popular, active, and has excellent feature
>support, including data visualization, traces, metrics, logs and so on, in
>particular, there is excellent support for data visualization
>
>
>3. Both have Java and c++ sdk that Doris can use
>
>—
>Best regards
>Stalary
>
Re: [Discuss][DSIP] Improve Doris observability
Posted by stalary <st...@163.com>.
Thank you very much, I will communicate with Luozenglin.
by stalary
On 05/16/2022 17:49，zhg yang<ya...@gmail.com> wrote：
Hi stalary
luozenglin <https://github.com/luozenglin> has done a lot of work on
OpenTelemetry and trace before, and has made some progress, I think you can
work with him to develop this thing together
Thanks
Yang Zhengguo
stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
I will split it for five steps, step1 will be visualized first.
Step1. Sql Plan
Step2. Execute Profile and data integration
Step3. Doris topology
Step4. Doris fe and be metrics
Step5. Doris log
by stalary
On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
Hi Stalary,
What kind of data will be visualized first?
Could you integrate profile data and sql plan together？
Any way, this is a good idea!
Minghong
At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
nice work！
------------------ Original ------------------
From: stalary <stalary@163.com>
Date: Sun,May 15,2022 7:13 PM
To: dev@doris.apache.org <dev@doris.apache.org>
Subject: Re: [Discuss][DSIP] Improve Doris observability
Dear All,
I’d like to improve Doris observability. This is achieved by introducing
Skywalking and OpenTelemetry. Parameters allow users to decide which
monitoring tool to use. First I want to support trace on the FE, implement
tracing of SQL statements.
The associated issue：https://github.com/apache/incubator-doris/issues/9577
Here are the results of my preliminary research
1. Currently OpenTelemetry is very convincing in terms of protocol
specifications, including traces, metrics, logs and so on, But it does not
provide native data visualization, we need to export data to Zipkin or
Prometheus to visualize.
2. Skywalking is currently popular, active, and has excellent feature
support, including data visualization, traces, metrics, logs and so on, in
particular, there is excellent support for data visualization
3. Both have Java and c++ sdk that Doris can use
—
Best regards
Stalary
Re: [Discuss][DSIP] Improve Doris observability
Posted by zhg yang <ya...@gmail.com>.
Hi stalary
luozenglin <https://github.com/luozenglin> has done a lot of work on
OpenTelemetry and trace before, and has made some progress, I think you can
work with him to develop this thing together
Thanks
Yang Zhengguo
stalary <st...@163.com> 于2022年5月15日周日 22:55写道：
> I will split it for five steps, step1 will be visualized first.
> Step1. Sql Plan
> Step2. Execute Profile and data integration
> Step3. Doris topology
> Step4. Doris fe and be metrics
> Step5. Doris log
>
>
> by stalary
>
>
> On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
> Hi Stalary,
> What kind of data will be visualized first?
> Could you integrate profile data and sql plan together？
> Any way, this is a good idea!
>
>
> Minghong
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
> nice work！
>
>
>
>
>
>
>
> ------------------ Original ------------------
> From: stalary <stalary@163.com>
> Date: Sun,May 15,2022 7:13 PM
> To: dev@doris.apache.org <dev@doris.apache.org>
> Subject: Re: [Discuss][DSIP] Improve Doris observability
>
>
>
>
> Dear All,
>
> I’d like to improve Doris observability. This is achieved by introducing
> Skywalking and OpenTelemetry. Parameters allow users to decide which
> monitoring tool to use. First I want to support trace on the FE, implement
> tracing of SQL statements.
>
> The associated issue：https://github.com/apache/incubator-doris/issues/9577
>
> Here are the results of my preliminary research
> 1. Currently OpenTelemetry is very convincing in terms of protocol
> specifications, including traces, metrics, logs and so on, But it does not
> provide native data visualization, we need to export data to Zipkin or
> Prometheus to visualize.
>
>
> 2. Skywalking is currently popular, active, and has excellent feature
> support, including data visualization, traces, metrics, logs and so on, in
> particular, there is excellent support for data visualization
>
>
> 3. Both have Java and c++ sdk that Doris can use
>
> —
> Best regards
> Stalary
>
Re: [Discuss][DSIP] Improve Doris observability
Posted by stalary <st...@163.com>.
I will split it for five steps, step1 will be visualized first.
Step1. Sql Plan
Step2. Execute Profile and data integration
Step3. Doris topology
Step4. Doris fe and be metrics
Step5. Doris log
by stalary
On 05/15/2022 22:27，Zhou Minghong<mi...@163.com> wrote：
Hi Stalary,
What kind of data will be visualized first?
Could you integrate profile data and sql plan together？
Any way, this is a good idea!
Minghong
At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
nice work！
------------------ Original ------------------
From: stalary <stalary@163.com>
Date: Sun,May 15,2022 7:13 PM
To: dev@doris.apache.org <dev@doris.apache.org>
Subject: Re: [Discuss][DSIP] Improve Doris observability
Dear All,
I’d like to improve Doris observability. This is achieved by introducing Skywalking and OpenTelemetry. Parameters allow users to decide which monitoring tool to use. First I want to support trace on the FE, implement tracing of SQL statements.
The associated issue：https://github.com/apache/incubator-doris/issues/9577
Here are the results of my preliminary research
1. Currently OpenTelemetry is very convincing in terms of protocol specifications, including traces, metrics, logs and so on, But it does not provide native data visualization, we need to export data to Zipkin or Prometheus to visualize.
2. Skywalking is currently popular, active, and has excellent feature support, including data visualization, traces, metrics, logs and so on, in particular, there is excellent support for data visualization
3. Both have Java and c++ sdk that Doris can use
—
Best regards
Stalary
Re:Re: [Discuss][DSIP] Improve Doris observability
Posted by Zhou Minghong <mi...@163.com>.
Hi Stalary,
What kind of data will be visualized first?
Could you integrate profile data and sql plan together？
Any way, this is a good idea!
Minghong
At 2022-05-15 19:26:05, "41108453" <41...@qq.com.INVALID> wrote:
>nice work！
>
>
>
>
>
>
>
>------------------ Original ------------------
>From: stalary <stalary@163.com>
>Date: Sun,May 15,2022 7:13 PM
>To: dev@doris.apache.org <dev@doris.apache.org>
>Subject: Re: [Discuss][DSIP] Improve Doris observability
>
>
>
>
>Dear All,
>
>I’d like to improve Doris observability. This is achieved by introducing Skywalking and OpenTelemetry. Parameters allow users to decide which monitoring tool to use. First I want to support trace on the FE, implement tracing of SQL statements.
>
>The associated issue：https://github.com/apache/incubator-doris/issues/9577
>
>Here are the results of my preliminary research
>1. Currently OpenTelemetry is very convincing in terms of protocol specifications, including traces, metrics, logs and so on, But it does not provide native data visualization, we need to export data to Zipkin or Prometheus to visualize.
>
>
>2. Skywalking is currently popular, active, and has excellent feature support, including data visualization, traces, metrics, logs and so on, in particular, there is excellent support for data visualization
>
>
>3. Both have Java and c++ sdk that Doris can use
>
>—
>Best regards
>Stalary
Re: [Discuss][DSIP] Improve Doris observability
Posted by 41108453 <41...@qq.com.INVALID>.
nice work！
------------------ Original ------------------
From: stalary <stalary@163.com>
Date: Sun,May 15,2022 7:13 PM
To: dev@doris.apache.org <dev@doris.apache.org>
Subject: Re: [Discuss][DSIP] Improve Doris observability
Dear All,
I’d like to improve Doris observability. This is achieved by introducing Skywalking and OpenTelemetry. Parameters allow users to decide which monitoring tool to use. First I want to support trace on the FE, implement tracing of SQL statements.
The associated issue：https://github.com/apache/incubator-doris/issues/9577
Here are the results of my preliminary research
1. Currently OpenTelemetry is very convincing in terms of protocol specifications, including traces, metrics, logs and so on, But it does not provide native data visualization, we need to export data to Zipkin or Prometheus to visualize.
2. Skywalking is currently popular, active, and has excellent feature support, including data visualization, traces, metrics, logs and so on, in particular, there is excellent support for data visualization
3. Both have Java and c++ sdk that Doris can use
—
Best regards
Stalary